We have the most recent versions of Sphinx (with Native Language Support), Pootle and Translate Toolkit running from checkout on our servers and a current checkout of the Sphinx project source in question. There is a directory for .pot files (message templates) built by Sphinx and one for .mo files (compiled catalogs) built from Pootle’s .po files (message catalogs). One last directory is used for translated HTML builds.

Attention: Apache should drop into the same system user as the one maintaining these files. This saves you a lot of hassle messing around with shared files, groups rights, yadda yadda.

Note: I will use Unix environment variable syntax for locations, feel free to statically insert your preferences there (we do).

Sphinx has to build message catalogs from a $SRC directory into Pootle’s PODIRECTORY set in localsettings.py. For our deployment, $PROJECT is always python.

sphinx-build -b gettext $SRC $PODIRECTORY/$PROJECT/templates

Pootle creates its translations in $PODIRECTORY/$PROJECT/$LANGUAGE but currently Sphinx only reads compiled message catalogs.

for file in $PODIRECTORY/$PROJECT/$LANGUAGE/*.po; do
    mofile="$MODIRECTORY/$LANGUAGE/LC_MESSAGES/$(basename $file | sed s/po$/mo/)"
    rm -f $mofile # msgfmt will do merging otherwise
    msgfmt "$file" -o "$mofile"
done

Sphinx can now pick the translations up from $MODIRECTORY by adding it to locale_dirs from your conf.py. The code snippet above only works for one (or non-overlapping) projects but is easily extended to multiple targets. (Warning: locale_dirs can not be set from the command line as it needs to be a list of paths).

sphinx-build -Eaq -Dlanguage=$LANGUAGE $SRC $BUILD/$LANGUAGE

Your HTML build should end up in $BUILD/$LANGUAGE.

NB. If you want to reproduce a Python documentation setup you will likely need to enable sphinx.ext.oldcmarkup for the time being because Python did not switch to Sphinx 1.0 as of the time of our deployment. This extension will become moot as soon as Python does the jump. Additionally you might want to enable AUTOSYNC and the Google Translate backend.

I am reprinting the full announcement made to the Python Documentation Special Interest Group here for posterity:

Dear Python Documentation community,

we are proud to announce the *BETA* launch of the translation services for the official Python documentation as part of Google’s Summer of Code. It is available from

http://pootle.python.org/

and is open for registration now. We have added a few languages that we felt would generate enough feedback but are always happy to add more language teams.

Please note that the software toolchain used to create this service is still experimental and might probably change substantially. We are trying our best to maintain stable services but cannot guarantee every bit you submit will be ultimately usable in our final translation targets.

If you are experiencing any trouble, want to bring up any suggestions or have other feedback do not hesitate to contact me or file a ticket at http://bitbucket.org/lehmannro/sphinx-i18n/issues.

Robert Lehmann

• a Sphinx extension to extract/incorporate translatable strings
• an interface to maintain translations

It turns out the latter half is already partially solved by Pootle and I can build on that instead of rolling my own half-baked ad-hoc web interface.

Now this changes my whole schedule (sigh) and I can concentrate on improving Pootle in the second term of the summer. We have a handful of priorities which will enable a pydotorg setup and wide-spread adoption of a Sphinx-Pootle conflation.

Installation

Pootle is easy_installable per se but behaves a bit strangely in virtual environments due to absolute paths in its setup. There are packages in Debian stable but they are old as the hills and not of any particular use as we expect changes to Pootle during this summer, aight. (I do not want to downplay Debian, and Pootle 2.x is already in the next release pool.)

Features

The most dramatic changes are going to be about Change Tracking. Pootle needs to grow facilities to show differences between updates, ie. several versions of message catalogs. Pootle already features checkers for a couple of metrics and should probably be extended by reStructuredText validation.

I have started discussion with the Pootle developers and they welcomed my plans.

With that done we have the translated message texts for each document node at our fingertips. These are still flat strings rather than nested document trees and thus not directly usable for our purposes.

As luck would have it Docutils readily exposes the reStructuredText parser and we can just turn the messages into small doctrees and merge those into our existing document. This is now implemented in Sphinx.

There is one last remaining caveat: the translation step as it stands turns the build process into a dog slow mess. Sphinx decided to update the whole environment (that is, re-read all files) when the language value changes; which is the case every. single. build. right now. For our current use I don’t care too hard though this needs to be fixed when sphinx-i18n is merged into upstream. It’s likely to require more in-depth architectural changes to Sphinx’ core to get this as quick and resource-friendly as possible.

Gentoo‘s tools remain mostly untranslated. There is no coordinated i18n effort and, if at all, they use plain PO files with a strong dependency on shared translation memory.

For software, FreeBSD follows the same strategy. They have a German manual in DocBook format which can be translated and pushed to version control. To maintain a uniform language style translators need to work on whole chapters. Issues are resolved through their mailing list which will supply SGML markup to plain text translations or pick up half-finished ones. Updates to the reference documentation are tracked through revision numbers.

The Sidux manual follows a similiar workflow: there is one person writing the reference documentation in English and one maintainer in charge per translation. Changes propagate by word of mouth.

Debian has the excellent Debian Description Translation Project (DDTP) which handles translations via e-mail and has a built-in notion of review. Their home-brewn web interface Debian Distributed Translation Server Satellite (DDTSS) acts as a gateway to their mail server.

Localization is a core value in the KDE culture. They have a lot of documentation on internationalization issues and try to remove friction where possible, eg. by encouraging semantic markup and context in translatable strings.
Approaching releases there are time spans where translatable strings ought not be changed by developers. This guarantees there are translations available on release date.

OpenSUSE has its translations for YaST and other system components stored in a central SVN repository. PO files are modified locally and aided by a shared memory. msgfmt statistics are adequately visualized.

I also had the chance to talk to Henning Eggers of Canonical at the Ubuntu BBQ who is part of the Launchpad Translations team. I learned that they are actually a pretty shallow layer above gettext message catalogs and settled for providing utilities such as shared translation memory around the actual process. Fuzzy translations are discarded because wrong translations are absolutely fatal to their use case. Henning mentioned they would be interested in change tracking (for instance by means of edit distance between two msgids) but do not have any such mechanisms in place currently.

I heard users absolutely want translation services to Just Work™ when they invest time in localization. Errors bubbling up from maintainers’ faults (such as  duplicate msgids) which inhibit their productivity usually kill motivation. Better diffs and change tracking seem to be a key feature people are looking for.

Docutils makes this as easy as shooting fish in a barrel — it provides nodes marked up as “containing text immediately” (nodes.TextElement) which just need to be serialized to messages.

Other projects such as Real World Haskell use the same policy for their comment spans. Of course paragraphs are the most trivial message. List items get a message each, as do admonitions.

The Translate Toolkit supplies a tool called posegment to split messages containing multiple sentences into smaller chunks but it has the usual fallacies like decomposing “Dr. Jekyll” into two messages.

“Raw messages?,” you say. That’s basically synonymous with woo, a lot of output… which is entirely useless. These messages contain no markup at all and thus are handy to estimate a message catalog’s contents and size but have no practical application for documents — except if you want to lose all inline markup, that is.

As the doctree is readily available during serialization to message catalogs I have looked into writers producing ReStructuredText themselves. I am pretty settled for reusing docutils’ reStructuredText parser, or at least its inliner, to rematerialize translations. There are a few subtle intricacies to inline markup which need to be resolved later on: some inline markup should be usable at the translator’s discretion (eg. strong, emphasis), other should not need to be retyped by every translator (eg. reference URIs, role targets) but could come in handy in some places (eg. references to multilingual pages).

I have briefly stumbled upon smaller style issues in the message catalogs while trying to rewrap msgids to 80 characters per line. Unfortunately — Daniel and Georg don’t seem too happy with this situation either — we are stuck with Python 2.4+ and thus do not have textwrap‘s newest features at our disposal. I consider this a non-issue for now though.

Georg supplied initial code review and instructed me to write tests and docs. He maintains a fork of sphinx-i18n for hotfixes and merging.

I abandoned the XLIFF format ­— as there really is no point in duplicate representation — and will focus my efforts on PO files. I am going to dive into gettext family of translation toolsuites to get an impression not only of the PO File Format Specification but of the whole normative landscape of solutions.

The new tool will require user feedback, user feedback, and user feedback and thus I require the community to help in building the best possible workflow for them. I will reach out to some projects in the future but appreciate contributions from anybody. (Py-Tutorial-de, I’m looking at you! hint hint)

I discussed the change tracking mechanism briefly with Martin and we concluded that walking the version history is a complex task which might require more time and skill than I have at hand. Thus it is desirable to save revision information explicitly in message catalogs.

I have been asked in advance why I chose gettext .PO files to store translations and we briefly discussed that issue. gettext was written for translating messages and not neccessarily whole paragraphs of free text. Its key tool used for updating translation sets, msgmerge, uses fuzzy matching with the intention of producing better results but this tends to fail. There is no notion of versioning and VCSes (or patch queues) need to be integrated into the workflow to retain history and inline markup is still a whole different can of worms (which I need to meditate about). The two selling points for .PO files are its suitability as a key-value store and that there are established tools after all. The other shortcomings have to be overcome by the new tool which will monitor version control and display stale documentation segments (and exports message catalogs along the way).

We had a look at Publican, a publishing system based on DocBook XML. For translations they use .PO files, too — it’s turtles all the way down.

It quickly became apparent that the tool for maintaining document translations will be separate from Sphinx’s core. There are already projects verging into the same direction, eg. Transifex / Txo or Launchpad Translations (which both neglect change tracking and are built for UI translations).

As there has to be at least one public instance on python.org we need to authenticate users. Invitations gained quite a bit of traction and give individual translation communities (groups) the possibility to self-organize. Integration with Roundup (Python’s bug tracker) would be nice-to-have but is not too important for now.

If you want to participate in our IRC meetings, we’ll be in #pocoo Wednesday evening.

I initially planned to file a simple patch but it grew into a full-fledged extension so the commit history is pretty meaningless now. Behind the scenes it still took about five iterations to get this Done Right.

I started off with writing a new builder from scratch which was a nice exercise for the i18n builder I am going to write but overachieving in terms of this simple extension. It was a pretty braindead implementation and used close to zero of Sphinx’s Builder API. I glimpsed briefly into inheriting from StandaloneHTMLBuilder — the HTML document builder — which was a little obtuse and clumsy for creating just a hand full of generic documents. With discovering the intricacies of the build process I learned how to hook into the right places and use existing build features (such as status iterators, which render a handy progressbar during the build).

It then hit me that the extension mechanism might well be a better entry point for my plugin and so it was. An initial version created all documents in isolation of the original build process which was okay but would’ve required a time machine for integrating the sidebar extension into existing pages, which struck me as an odd and overly heavy dependency. I quickly split up the (already two-pass) build process into a collection and a build phase and could easily insert links to the sidebar panels into original documents.

Lessons Learned

• Pocoo’s already using Mercurial. Developing features, as minor as they may seem, should happen in a fork, not a checkout-patch (which I’m still used to from Python SVN).
• Sphinx’ documentation needs update in the “Writing a new builder” chapter. I will perhaps come around to that while writing/documenting my next builder.
• Review is awesome and does not need an extensive toolchain. Georg Brandl briefly discussed the extension (by loading and trying it himself); his comments allowed me to fill in gaps in the documentation.
• Docutils perhaps has a few bugs in trunk that interfered during debugging. (See #2993967 and r6293/#2993967).