John's Blog

0.5.2 is now available for download! This is only a bug fix release but it fixes the “add existing file” feature that broke with 0.5.1.

There is no Linux x64 build at this time because my build machine died while building this package. I’ll be looking into options to replace this machine but Linux x64 users will have to make due with the x86 Linux release for the time being. I’ve been having problems with my build machine (since the 0.5.0 release) but this time it looks like it’s might be beyond reviving.

Tags: release, Sigil.

This is primarily a maintenance release with a large number of fixes. Find and replace (F&R), GUI tweaks and spell check were the major focus. Highlights include: F&R now supports wrapping and the ability to search within selected files. The GUI now (hopefully) remembers cursor position when switching views. en_GB dictionary was added, all dictionaires were updated and hyphenation dictionaries are now included by default. For a full list of all changes for this release please see the Changelog.

This release wouldn’t have come off as smoothly, quickly or with as many fixes and new features without the help of meme from MobileRead. Thank you for the time and effort you put into making this release possible.

Starting with this release I want to start making smaller releases with less time between them. I plan to target approximately 4 weeks between releases give or take two weeks. I believe this will make development more fluid and allow for greater user input.

My short term plans for upcoming releases are (not all of these will be in the next release):

• Make book view editing more robust.
• Firebug style element inspection.
• Make code view editing more robust.
• Plugin interface.
• Move auto cleaning and restructuring into plugins.

My long term plans are EPUB 3. This is not a short term goal but I plan / want to make this a reality by the end of this year.

Tags: release, Sigil.

There have been some confusion about how find works in, the now released, 0.5.0. The confusion stems from the the 0.4.90x betas. One method was used in the early betas and it was changed later on. This all stems from the regular expression engine being changed from QRegExp to PCRE. The issue at hand is how and when the cursor is taken info account when running a find. In this regard 0.5.0 works no different than 0.4.2.

When doing a count the cursor is ignored. The entire document is taken into account from start to end.

When doing a find next the find starts from the cursor location. Everything before the cursor is ignored and not taken into account. This can, in some cases when using a regular expression, lead to the number of matches being different from the total returned by count. Again, this can only happen when using a regular expression. The reason is a regular expression can have matches that match the expression within a single match. For example:

Expression:

<div>.+</div>

Text:

<div>blah <div> blah </div> blah </div>

The expression will match the text from beginning to end. If you put the cursor to the right of the first < then the math will start from the second div and go to the end. This is because regular expressions can match a variable amount of text. Unlike a fixed expression like “abc” which will always match “abc”.

Finding backwards will match from the start of the document up until the cursor position. This is done by finding all matches from the start to the cursor then using the last match. Again, in the case of regular expressions, a backward find can match different text than a forward find.

Find forward and backward find from the cursor so its position in the document taken into account. In the majority of instances find backward, forward and count will all match the same exact text. However, it is possible, due to their nature, to construct a regular expression that can match differently segments of text within a segment of text depending on where the cursor is located.

The above also applies to replace as a find is run to find the text to replace.

Tags: Sigil.

For some time now Sigil has had two different help forums. One at MobileRead and the other as a Google Group. This has caused quite a bit of confusion because people don’t know the best place to go for help.

I’ve decided to close (it’s already done so don’t ask me to reconsider) the Google Group. This was an easy decision because 1) MobileRead’s sub forum gets more traffic, 2) I use MobileRead and I don’t use the Google Gorup, 3) Most posts on the Google Group were unanswered making it a poor place to go for help.

Tags: Sigil.

I’m happy to announce the release of Sigil version 0.5.0.

0.5.0 comes with a number of bug fixes and some major new features:

• Inline spell check in code view
• Support for PCRE in search and replace
• Translations into 15 languages

Please see the changelog for a full list of changes in this release.

One smaller change is I’ve decided to drop OpenCandy. Surprisingly I’ve only encountered one complaint about OpenCandy and it was directed at bundling offers for other software inside of an installer not at OpenCandy in particular. I want to make it clear that this change is not due to user request or opposition to OpenCandy but my own decision.

While I respect what OpenCandy the company is doing and I don’t see anything wrong with the offerings they provide I don’t think their system is right for Sigil. The big thing I don’t like is OpenCandy’s installer components are distributed as closed source binary modules. While my understanding is this can be used without running afoul of the GPL I fully believe it goes against the spirit of the GPL and open source in general.

Thank you to everyone who provided feedback and helped during the beta process.

Tags: release, Sigil.

Introduction

A question I often get asked is what kind of workflow is best for creating ebooks. The short answer is there is no best, perfect, or standard way to create ebooks. There are as many workflows as there are people. I have regular contact with a comercial company (Booknook.biz) that provides digital file to ebook creation servies (I am not affiliated with or work for this company). The company is broken down into teams and each team has a very different work flow, uses very different tools and they all produce the same quality (high) ebooks. Suffice it to say there is no one way that works best. What I want to outline is a few different ways that I have seen work well for people in common situations.

The big thing to think about is what you want to do with the final look of the ebook. There are applications like Jutoh and Atlantis Word Processor that bill themselves as the equivalent of Word for ebook creation. Atlantis even goes as far as being able to directly open Word documents. These two are not free but compared to the cost of Word itself they are not over priced. These are tools with strong WYSIWYG formatting support.

Sigil fits into the ebook creation picture in cases where a high degree of control is desired. While Sigil does have some WYSIWYG features, it excels at working on code that comprises an ePub internally. Sigil is very much for manual, do it yourself, or fine grain control needs. That said, many users of Sigil use it solely for its WYSIWYG features. Sigil also has the advantage of being free (price) and open source.

In all cases I recommend getting your book into the ePub format. As noted there are many tools for working with ePub. The internals of the format are HTML and CSS which means it’s easy to learn how to do advanced formatting. Also, ePub is an open, fully documented, and standardized format. Further, it’s very easy to convert an ePub into a variety of other ebook formats such as mobi for use with Amazon’s Kindle.

Authors starting with a Word document who don’t want to know about HTML or CSS that makes up the inside of an ePub

Often times an author has already written their book. They have it saved as a Word document or in some other similar format. The challenge in this case is going from their favored text editor to an ebook. In this case I recommend either copy and pasting the text from Word into Jutoh, Sigil or opening the Word document with Atlantis Word Processor. Use the inbuilt features to add the desired formatting and save as an ePub.

This works and can produce a functional ebook. However you are limited by what you can do with the formatting and dealing with quirks in various reading systems.

The case for manually working on an ePub’s internals

Using Sigil to actually work on the internals of the ePub allow for quirks in different reading systems to be accommodated. Two major examples I can think of that illustrate this are thumbnail images on the Nook Touch and the Kobo iPad/iPod apps. Both have very specific requirements in order for them to appear properly. Calibre, for example, takes issues like these into account when it creates an ePub but this doesn’t mean calibre handles every quirk in every reader out there. Manual intervention is sometimes required. Also, this goes back to professional digitization services (like I mentioned earlier) because they know about the little quirks of the most popular reading devices and can ensure an ebook looks the same across all devices.

Three stage ebook creation

What I can say is the best processes in my experience when using Sigil is three stage: write, convert, finish.

1. Write the book in what ever format you’re most confortable using what ever tools you’re most confortable with. The caveat is you really need to be working in a digital medium. At some point you will have to digitize and scanning in pages of paper then correcting errors is going to be a long and arduous process. In most cases simply typing your book into Word is going to save you time after you’ve finished writing if you go the pen and paper route. Most authors are going to use Word which lets them, write, edit and do basic formatting. At this point leave the formatting pretty basic.
2. Use an automated conversion tool like calibre to get your book into a format that is a bit more editable as an ebook. Word is a great editor but Word files are complex (and older versions do not have official format specifications). For Word the best advise is to have Word save as HTML. This will lead to it’s own issues but they can be worked around. HTML produced by Word will have basic formatting intact and the beauty of using an automated conversion tool is a lot will come through when converting to ePub.
3. Take that mostly formatted ePub put it into Sigil and make the necessary changes to make it look perfect. Since Sigil allows for direct access to the XHTML and CSS you can use a number of different techniques to ensure it looks exactly how you want. Also, if you are working on a number of book and want to ensure consistant formatting you can do things like easily use a default stylesheet. The best way I can sum up how the majority of people use and how I view Sigil is, Sigil is for people who know XHTML, CSS, and regular expressions and want to use their knowledge.

Three stage variation: Markdown and Textile with plain text

I personally use a slight variation of the above. I prefer to use plain text with either Markdown or Textile for basic formatting. Then conversion with calibre and final adjustments with Sigil. I use both depending on what I want to accomplish as I find Markdown more intuitive but Textile allows for things like margins and text alignment. Even without using Markdown or Textile converting using calibre with heuristic processing will get most novels about 75-90% formatted.

Three stage variation: Starting with HTML and CSS

The biggest difference with this variation is it skips right to step three. It is possible to still use calibre to take the HTML and produce an ePub but in this case I would advise against doing so. calibre is an automated tool and will make changes to the HTML and CSS. Visually it should look the same but the HTML and CSS will be different. This can be good if you have HTML and CSS that is so horrible you need it cleaned as calibre does some normalization but if you’re using your own hand crafted HTML and CSS you probably don’t want it modified.

Once you have all of your files ready, open Sigil and have a blank document open. Right click in the book browser on the text folder. Choose the Add Existing Files. Select your HTML files and import them. You should now see the Text section includes all of those files. Do the same with your CSS.

You many need to edit the HTML files to ensure that the CSS is referenced from the correct location. All stylesheets are located in the Styles directory which is one step above the Text directory. For example:

<head>
<link href="../Styles/stylesheet.css" rel="stylesheet" type="text/css">
</head>

If you have any images add them in the same way you added the HTML and CSS files. Also, make sure they’re referenced properly just like you did with the CSS.

Check the look in the book view to make sure everything looks correct. Also don’t forget to delete the Section0001.xhtml that is left over from it being a new file.

As you added each file the filename is perserved and it is added to the TOC based on the file itself. You have two options for working with the TOC. If you want to leave it file based you’re pretty much done. If you don’t like the text that displays in the TOC you can edit the toc.ncx and change the text.

You’re other option is to generate the TOC from headings. In the TOC browser (should be on the right hand side) click generate TOC from headings. Automatic TOC generation is based on h1 – h3 tags. h2 are nested under h1 and h3 are nested under h2. The only other way to deal with the TOC is to manually edit the toc.ncx by hand.

Conclusion

What I would recommend to authors is take the easiest path. If you’re  happy with the results of using Jutoh or Atlantis then your done. If you want to get a bit more involved use a variation of the three stage work flow I outlined. If all this seems like too much to deal with, pay someone to do it for you.

.

The first beta for 0.5 (0.4.902) is now available.

There are a few new features I’m most interested in getting feedback on. Inline spell check, translations, and the new PCRE engine. Of course crashes and major issues will be looked into and hopefully fixed before the final release.

.

The majority of the data loss issues have been mitigated at this point. With a work flow of open, save as after major changes and saving after minor ones, catastrophic data loss can be worked around to the point that Sigil can and is being used on a day to day basis.

That said, there are issues with data loss in Sigil and they are a priority. I’m currently finishing up the 0.5 release (I do not have a set release date at this point) which is mainly a feature release and only addresses some of the the data loss issue. For example you can still have everything in an entire XHTML document removed by putting a malformed XML header in the document.

The issue has three components that require major work to fix. I hope to have it all completed for the 0.6 release but it’s going to be some time it’s ready.

The issues are:

1) Sigil currently uses Tidy to clean all XHTML to ensure it conforms (as much as it can) to the XHTML spec. I have seen Tidy remove tags it thinks are empty when they influence how the document is rendered. I want to keep Tidy as part of Sigil but I believe it should only be run when the user asks for it and any changes it makes the user should be able to revert.

2) An intermediate data store is used that requires valid XML is used. This store shuffles data between the book and code view. Due to this store requiring valid XML (valid XHTML conforms) there is the potential for data loss if it has to auto correct the XHTML. If you are in code view and have malformed structural issues with the XHTML and move out of it there is a warning dialog. This only appears when you are working on one file at a time. If you are replacing across multiple files auto correction is used and this can lead to data loss. This data store needs to be replaced with one that does not require valid XML.

3) Putting malformed content into the book view will cause the book view to try to correct it. Again auto correction can lead to data loss. This is mitigated by the malformed error dialog but many users just disable it and find that sections of their document are missing after looking at it in book view. Also, the book view is a WYSIWYG tool so it does make structural changes to the document and these may or may not be what the user expects. As with Tidy changes made by the book view need to be able to be reverted. I am thinking about ways to make the fact that the book view more obvious that it makes changes to the document. This way the user is aware that they need to use undo (doesn’t currently work for book view changes) to revert the changes if they don’t like them. I’m thinking about using a preview mode by default that doesn’t make any changes and an edit mode to make this distinction obvious.

The above issues can be fixed but they are not quick or easy changes. I plan on making them for the 0.6 release as part of the changes necessary to support EPUB 3. However, there is the possibility that they will slip to 0.7 due to how large they are. Unfortunately, all I can say right now is I’m aware of the issue, I know what the cause is, and I have an idea of how to correct it but it’s not going to happen tomorrow.

Tags: Sigil.

I have been working on adding inline spell check to Sigil recently and ran into a quirk on Qt that isn’t immediately obvious. I ended up having to look though the Qt source code to understand exactly what was happening.

When dealing with a QPlainTextEdit you can get the QTextCursor and use the charFormat() function to retrieve the QTextCharFormat for the character before the cursor. This does not work when the formatting is set by a QSyntaxHighlighter!.

charFormat retrieves the character format that has explicitly been set on the QPlainTextEdit. QSyntaxHighlighter does not directly set the formatting on the QPlainTextEdit. Instead QSyntaxHighlighter sets the format in additionalFormats as part of the block layout. All formatting for the block the cursor is currently in can be accessed by using QPlainTextEdit::textCursor().block().layout()->additionalFormats().

QTextLayout::additionalFormats() returns a list of FormatRange objects. A FormatRange gives the start of the formatting (relative to the block not the full text in the QPlainTextEdit), the length and the formatting (as set by the QSyntaxHighlighter). Simply loop over all of the FormatRange objects and check if the cursor is within a range to determine what formatting is applied to a particular part of the block’s text. Use QTextCursor::positionInBlock() to determine the relative position of the cursor within the block.

Here is an example from Sigil that I use for spell checking. It determines if a particular segment of text has the misspelled word style applied to it. It then selects the text.

QTextCursor c = textCursor();
int pos = c.positionInBlock();
foreach (QTextLayout::FormatRange r, textCursor().block().layout()->additionalFormats()) {
    if (pos >= r.start && pos <= r.start + r.length && r.format.underlineStyle() == QTextCharFormat::SpellCheckUnderline) {
        c.setPosition(c.block().position() + r.start);
        c.movePosition(QTextCursor::Right, QTextCursor::KeepAnchor, r.length);
        setTextCursor(c);
        break;
    }
}

*Note: QTextEdit can be substituted any place QPlainTextEdit is used. This applies to both not just QPlainTextEdit.

Tags: c++, qt.

One of the the new features that has been implemented for 0.5 (release date yet to be determined) is support for Translations. For Sigil’s first supported language Grzegorz Wolszczak has provided a Polish translation. Currently translations are loaded based upon the current system locale. There no support for choosing the language via preferences. This may come at a later time but for now I believe that using the system locale will handle the majority of user needs.

I’ve put together a wiki page with instructions for creating translations. This first revision is a bit basic but as people have questions I plan to update it to make it more robust.

Tags: internationalization, language, qt, Sigil, translations.