Update the i18n_subsites plugin, addresses many issues

Major highlights
................
- fixed and improved cross-linking (fixes #333) with URLs
  containing e.g. localized month names
  (thanks to issue getpelican/pelican#1198)
- support for custom ``SITEURL`` and ``OUTPUT_PATH`` hierarchy
  (fixes #182)
- sharing of static files (including those of the theme) among
  subsites (fixes #180)

Technical highlights
....................
- added a test suite (works with pelican 3.4)
- translations are installed into Jinja2 environments of all
  generators
- old locale is restored after generation, fixes autoreload

The documentation has been updated and improved (mostly in terms of
formatting).

Known issues
............
- due to the redesign required for correct cross-linking, older
  versions of Pelican (<3.4) are not supported, because they lack
  certain signals
- the ``HIDE_UNTRANSLATED_CONTENT`` setting has been deprecated in
  favor of the ``I18N_UNTRANSLATED_{ARTICLES,PAGES}`` settings which
  offer more control in order to fix #211.
- the test suite works only with pelican 3.4, later versions add a
  timezone field to the date

README.rst

@@ -1,73 +1,156 @@
-======================
+=======================
  I18N Sub-sites Plugin
-======================
+=======================
 
-This plugin extends the translations functionality by creating internationalized sub-sites for the default site. It is therefore redundant with the *\*_LANG_{SAVE_AS,URL}* variables, so it disables them to prevent conflicts.
+This plugin extends the translations functionality by creating
+internationalized sub-sites for the default site.
+
+This plugin is designed for Pelican 3.4 and later.
 
 What it does
 ============
-1. The *\*_LANG_URL* and *\*_LANG_SAVE_AS* variables are set to their normal counterparts (e.g. *ARTICLE_URL*) so they don't conflict with this scheme.
-2. While building the site for *DEFAULT_LANG* the translations of pages and articles are not generated, but their relations to the original content is kept via links to them.
-3. For each non-default language a "sub-site" with a modified config [#conf]_ is created [#run]_, linking the translations to the originals (if available). The configured language code is appended to the *OUTPUT_PATH* and *SITEURL* of each sub-site. For each sub-site, *DEFAULT_LANG* is changed to the language of the sub-site so that articles in a different language are treated as translations.
 
-If *HIDE_UNTRANSLATED_CONTENT* is True (default), content without a translation for a language is generated as hidden (for pages) or draft (for articles) for the corresponding language sub-site.
+1. When the content of the main site is being generated, the settings
+   are saved and the generation stops when content is ready to be
+   written. While reading source files and generating content objects,
+   the output queue is modified in certain ways:
 
-.. [#conf] For each language a config override is given in the *I18N_SUBSITES* dictionary.
+  - translations that will appear as native in a different (sub-)site
-.. [#run] Using a new *PELICAN_CLASS* instance and its ``run`` method, so each sub-site could even have a different *PELICAN_CLASS* if specified in *I18N_SUBSITES* conf overrides.
+    will be removed
+  - untranslated articles will be transformed to drafts if
+    ``I18N_UNTRANSLATED_ARTICLES`` is ``'hide'`` (default), removed if
+    ``'remove'`` or kept as they are if ``'keep'``.
+  - untranslated pages will be transformed into hidden pages if
+    ``I18N_UNTRANSLATED_PAGES`` is ``'hide'`` (default), removed if
+    ``'remove'`` or kept as they are if ``'keep'``.''
+  - additional content manipulation similar to articles and pages can
+    be specified for custom generators in the ``I18N_GENERATOR_INFO``
+    setting.
+
+2. For each language specified in the ``I18N_SUBSITES`` dictionary the
+   settings overrides are applied to the settings from the main site
+   and a new sub-site is generated in the same way as with the main
+   site until content is ready to be written.
+3. When all (sub-)sites are waiting for content writing, all removed
+   contents, translations and static files are interlinked across the
+   (sub-)sites.
+4. Finally, all the output is written.
 
 Setting it up
 =============
 
-For each extra used language code, a language-specific variables overrides dictionary must be given (but can be empty) in the *I18N_SUBSITES* dictionary
+For each extra used language code, a language-specific settings overrides
+dictionary must be given (but can be empty) in the ``I18N_SUBSITES`` dictionary
 
 .. code-block:: python
 
     PLUGINS = ['i18n_subsites', ...]
 
-    # mapping: language_code -> conf_overrides_dict
+    # mapping: language_code -> settings_overrides_dict
     I18N_SUBSITES = {
         'cz': {
 	    'SITENAME': 'Hezkej blog',
 	    }
 	}
 
-- The language code is the language identifier used in the *lang* metadata. It is appended to *OUTPUT_PATH* and *SITEURL* of each I18N sub-site.
+Default and special overrides
-- The internationalized config overrides dictionary may specify configuration variable overrides — e.g. a different *LOCALE*, *SITENAME*, *TIMEZONE*, etc. However, it **must not** override *OUTPUT_PATH* and *SITEURL* as they are modified automatically by appending the language code.
+-----------------------------
+The settings overrides may contain arbitrary settings, however, there
+are some that are handled in a special way:
+
+``SITEURL``
+  Any overrides to this setting should ensure that there is some level
+  of hierarchy between all (sub-)sites, because Pelican makes all URLs
+  relative to ``SITEURL`` and the plugin can only cross-link between
+  the sites using this hierarchy. For instance, with the main site
+  ``http://example.com`` a sub-site ``http://example.com/de`` will
+  work, but ``http://de.example.com`` will not. If not overridden, the
+  language code (the language identifier used in the ``lang``
+  metadata) is appended to the main ``SITEURL`` for each sub-site.
+``OUTPUT_PATH``, ``CACHE_PATH``
+  If not overridden, the language code is appended as with ``SITEURL``.
+  Separate cache paths are required as parser results depend on the locale.
+``STATIC_PATHS``, ``THEME_STATIC_PATHS``
+  If not overridden, they are set to ``[]`` and all links to static
+  files are cross-linked to the main site.
+``THEME``, ``THEME_STATIC_DIR``
+  If overridden, the logic with ``THEME_STATIC_PATHS`` does not apply.
+``DEFAULT_LANG``
+  This should not be overridden as the plugin changes it to the
+  language code of each sub-site to change what is perceived as translations.
 
 Localizing templates
 --------------------
 
-Most importantly, this plugin can use localized templates for each sub-site. There are two approaches to having the templates localized:
+Most importantly, this plugin can use localized templates for each
+sub-site. There are two approaches to having the templates localized:
 
-- You can set a different *THEME* override for each language in *I18N_SUBSITES*, e.g. by making a copy of a theme ``my_theme`` to ``my_theme_lang`` and then editing the templates in the new localized theme. This approach means you don't have to deal with gettext ``*.po`` files, but it is harder to maintain over time.
+- You can set a different ``THEME`` override for each language in
-- You use only one theme and localize the templates using the `jinja2.ext.i18n Jinja2 extension <http://jinja.pocoo.org/docs/templates/#i18n>`_. For a kickstart read this `guide <./localizing_using_jinja2.rst>`_.
+  ``I18N_SUBSITES``, e.g. by making a copy of a theme ``my_theme`` to
+  ``my_theme_lang`` and then editing the templates in the new
+  localized theme. This approach means you don't have to deal with
+  gettext ``*.po`` files, but it is harder to maintain over time.
+- You use only one theme and localize the templates using the
+  `jinja2.ext.i18n Jinja2 extension
+  <http://jinja.pocoo.org/docs/templates/#i18n>`_. For a kickstart
+  read this `guide <./localizing_using_jinja2.rst>`_.
 
-It may be convenient to add language buttons to your theme in addition to the translation links of articles and pages. These buttons could, for example, point to the *SITEURL* of each (sub-)site. For this reason the plugin adds these variables to the template context:
+Additional context variables
+............................
 
-main_lang
+It may be convenient to add language buttons to your theme in addition
-  The language of the top-level site — the original *DEFAULT_LANG*
+to the translation links of articles and pages. These buttons could,
-main_siteurl
+for example, point to the ``SITEURL`` of each (sub-)site. For this
-  The *SITEURL* of the top-level site — the original *SITEURL*
+reason the plugin adds these variables to the template context:
-lang_siteurls
-  An ordered dictionary, mapping all used languages to their *SITEURL*. The ``main_lang`` is the first key with ``main_siteurl`` as the value. This dictionary is useful for implementing global language buttons that show the language of the currently viewed (sub-)site too.
-extra_siteurls
-  An ordered dictionary, subset of ``lang_siteurls``, the current *DEFAULT_LANG* of the rendered (sub-)site is not included, so for each (sub-)site ``set(extra_siteurls) == set(lang_siteurls) - set([DEFAULT_LANG])``. This dictionary is useful for implementing global language buttons that do not show the current language.
 
-If you don't like the default ordering of the ordered dictionaries, use a Jinja2 filter to alter the ordering.
+``main_lang``
+  The language of the main site — the original ``DEFAULT_LANG``
+``main_siteurl``
+  The ``SITEURL`` of the main site — the original ``SITEURL``
+``lang_siteurls``
+  An ordered dictionary, mapping all used languages to their
+  ``SITEURL``. The ``main_lang`` is the first key with ``main_siteurl``
+  as the value. This dictionary is useful for implementing global
+  language buttons that show the language of the currently viewed
+  (sub-)site too.
+``extra_siteurls``
+  An ordered dictionary, subset of ``lang_siteurls``, the current
+  ``DEFAULT_LANG`` of the rendered (sub-)site is not included, so for
+  each (sub-)site ``set(extra_siteurls) == set(lang_siteurls) -
+  set([DEFAULT_LANG])``. This dictionary is useful for implementing
+  global language buttons that do not show the current language.
+``relpath_to_site``
+  A function that returns a relative path from the first (sub-)site to
+  the second (sub-)site where the (sub-)sites are identified by the
+  language codes given as two arguments.
 
-This short `howto <./implementing_language_buttons.rst>`_ shows two example implementations of language buttons.
+If you don't like the default ordering of the ordered dictionaries,
+use a Jinja2 filter to alter the ordering.
+
+All the siteurls above are always absolute even in the case of
+``RELATIVE_URLS == True`` (it would be to complicated to replicate the
+Pelican internals for local siteurls), so you may rather use something
+like ``{{ SITEURL }}/{{ relpath_to_site(DEFAULT_LANG, main_lang }}``
+to link to the main site.
+
+This short `howto <./implementing_language_buttons.rst>`_ shows two
+example implementations of language buttons.
 
 Usage notes
 ===========
-- It is **mandatory** to specify *lang* metadata for each article and page as *DEFAULT_LANG* is later changed for each sub-site, so content without *lang* metadata woudl be rendered in every (sub-)site.
+- It is **mandatory** to specify ``lang`` metadata for each article
-- As with the original translations functionality, *slug* metadata is used to group translations. It is therefore often convenient to compensate for this by overriding the content URL (which defaults to slug) using the *URL* and *Save_as* metadata.
+  and page as ``DEFAULT_LANG`` is later changed for each sub-site, so
-
+  content without ``lang`` metadata would be rendered in every
-Future plans
+  (sub-)site.
-============
+- As with the original translations functionality, ``slug`` metadata
-
+  is used to group translations. It is therefore often convenient to
-- add a test suite
+  compensate for this by overriding the content URL (which defaults to
+  slug) using the ``url`` and ``save_as`` metadata. You could also
+  give articles e.g. ``name`` metadata and use it in ``ARTICLE_URL =
+  '{name}.html'``.
 
 Development
 ===========
 
-- A demo and test site is in the ``gh-pages`` branch and can be seen at http://smartass101.github.io/pelican-plugins/
+- A demo and a test site is in the ``gh-pages`` branch and can be seen
+  at http://smartass101.github.io/pelican-plugins/

_regenerate_context_helpers.py

@@ -1,81 +0,0 @@
-
-import math
-import random
-from collections import defaultdict
-from operator import attrgetter, itemgetter
-
-
-def regenerate_context_articles(generator):
-    """Helper to regenerate context after modifying articles draft state
-
-    essentially just a copy from pelican.generators.ArticlesGenerator.generate_context
-    after process_translations up to signal sending
-
-    This has to be kept in sync untill a better solution is found
-    This is for Pelican version 3.3.0
-    """
-    # Simulate __init__ for fields that need it
-    generator.dates = {}
-    generator.tags = defaultdict(list)
-    generator.categories = defaultdict(list)
-    generator.authors = defaultdict(list)
-    
-
-    # Simulate ArticlesGenerator.generate_context 
-    for article in generator.articles:
-        # only main articles are listed in categories and tags
-        # not translations
-        generator.categories[article.category].append(article)
-        if hasattr(article, 'tags'):
-            for tag in article.tags:
-                generator.tags[tag].append(article)
-        # ignore blank authors as well as undefined
-        if hasattr(article, 'author') and article.author.name != '':
-            generator.authors[article.author].append(article)
-
-
-    # sort the articles by date
-    generator.articles.sort(key=attrgetter('date'), reverse=True)
-    generator.dates = list(generator.articles)
-    generator.dates.sort(key=attrgetter('date'),
-            reverse=generator.context['NEWEST_FIRST_ARCHIVES'])
-
-    # create tag cloud
-    tag_cloud = defaultdict(int)
-    for article in generator.articles:
-        for tag in getattr(article, 'tags', []):
-            tag_cloud[tag] += 1
-
-    tag_cloud = sorted(tag_cloud.items(), key=itemgetter(1), reverse=True)
-    tag_cloud = tag_cloud[:generator.settings.get('TAG_CLOUD_MAX_ITEMS')]
-
-    tags = list(map(itemgetter(1), tag_cloud))
-    if tags:
-        max_count = max(tags)
-    steps = generator.settings.get('TAG_CLOUD_STEPS')
-
-    # calculate word sizes
-    generator.tag_cloud = [
-        (
-            tag,
-            int(math.floor(steps - (steps - 1) * math.log(count)
-                / (math.log(max_count)or 1)))
-        )
-        for tag, count in tag_cloud
-    ]
-    # put words in chaos
-    random.shuffle(generator.tag_cloud)
-
-    # and generate the output :)
-
-    # order the categories per name
-    generator.categories = list(generator.categories.items())
-    generator.categories.sort(
-            reverse=generator.settings['REVERSE_CATEGORY_ORDER'])
-
-    generator.authors = list(generator.authors.items())
-    generator.authors.sort()
-
-    generator._update_context(('articles', 'dates', 'tags', 'categories',
-                          'tag_cloud', 'authors', 'related_posts'))
-    

i18n_subsites.py

@@ -1,189 +1,440 @@
-"""i18n_subsites plugin creates i18n-ized subsites of the default site"""
+"""i18n_subsites plugin creates i18n-ized subsites of the default site
 
+This plugin is designed for Pelican 3.4 and later
+"""
 
 
 import os
 import six
 import logging
+import posixpath
+
+from copy import copy
 from itertools import chain
-from collections import defaultdict, OrderedDict
+from operator import attrgetter
+from collections import OrderedDict
+from contextlib import contextmanager
+from six.moves.urllib.parse import urlparse
 
 import gettext
+import locale
 
 from pelican import signals
-from pelican.contents import Page, Article
+from pelican.generators import ArticlesGenerator, PagesGenerator
 from pelican.settings import configure_settings
-
+from pelican.contents import Draft
-from ._regenerate_context_helpers import regenerate_context_articles
-
 
 
 # Global vars
-_main_site_generated = False
+_MAIN_SETTINGS = None     # settings dict of the main Pelican instance
-_main_site_lang = "en"
+_MAIN_LANG = None         # lang of the main Pelican instance
-_main_siteurl = ''
+_MAIN_SITEURL = None      # siteurl of the main Pelican instance
-_lang_siteurls = None
+_MAIN_STATIC_FILES = None # list of Static instances the main Pelican instance
-logger = logging.getLogger(__name__)
+_SUBSITE_QUEUE = {}   # map: lang -> settings overrides
+_SITE_DB = OrderedDict()           # OrderedDict: lang -> siteurl
+_SITES_RELPATH_DB = {}       # map: (lang, base_lang) -> relpath
+# map: generator -> list of removed contents that need interlinking
+_GENERATOR_DB = {}
+_NATIVE_CONTENT_URL_DB = {} # map: source_path -> content in its native lang
+_LOGGER = logging.getLogger(__name__)
 
 
+@contextmanager
+def temporary_locale(temp_locale=None):
+    '''Enable code to run in a context with a temporary locale
 
-def disable_lang_vars(pelican_obj):
+    Resets the locale back when exiting context.
-    """Set lang specific url and save_as vars to the non-lang defaults
+    Can set a temporary locale if provided
-
+    '''
-    e.g. ARTICLE_LANG_URL = ARTICLE_URL
+    orig_locale = locale.setlocale(locale.LC_ALL)
-    They would conflict with this plugin otherwise
+    if temp_locale is not None:
-    """
+        locale.setlocale(locale.LC_ALL, temp_locale)
-    global _main_site_lang, _main_siteurl, _lang_siteurls
+    yield
-    s = pelican_obj.settings
+    locale.setlocale(locale.LC_ALL, orig_locale)
-    for content in ['ARTICLE', 'PAGE']:
-        for meta in ['_URL', '_SAVE_AS']:
-            s[content + '_LANG' + meta] = s[content + meta]
-    if not _main_site_generated:
-        _main_site_lang = s['DEFAULT_LANG']
-        _main_siteurl = s['SITEURL']
-        _lang_siteurls = [(lang, _main_siteurl + '/' + lang) for lang in s.get('I18N_SUBSITES', {}).keys()]
-        # To be able to use url for main site root when SITEURL == '' (e.g. when developing)
-        _lang_siteurls = [(_main_site_lang, ('/' if _main_siteurl == '' else _main_siteurl))] + _lang_siteurls
-        _lang_siteurls = OrderedDict(_lang_siteurls)
 
 
+def initialize_dbs(settings):
+    '''Initialize internal DBs using the Pelican settings dict
 
-def create_lang_subsites(pelican_obj):
+    This clears the DBs for e.g. autoreload mode to work
-    """For each language create a subsite using the lang-specific config
+    '''
+    global _MAIN_SETTINGS, _MAIN_SITEURL, _MAIN_LANG, _SUBSITE_QUEUE
+    _MAIN_SETTINGS = settings
+    _MAIN_LANG = settings['DEFAULT_LANG']
+    _MAIN_SITEURL = settings['SITEURL']
+    _SUBSITE_QUEUE = settings.get('I18N_SUBSITES', {}).copy()
+    prepare_site_db_and_overrides()
+    # clear databases in case of autoreload mode
+    _SITES_RELPATH_DB.clear()
+    _NATIVE_CONTENT_URL_DB.clear()
+    _GENERATOR_DB.clear()
 
-    for each generated lang append language subpath to SITEURL and OUTPUT_PATH
+
-    and set DEFAULT_LANG to the language code to change perception of what is translated
+def prepare_site_db_and_overrides():
-    and set DELETE_OUTPUT_DIRECTORY to False to prevent deleting output from previous runs
+    '''Prepare overrides and create _SITE_DB
-    Then generate the subsite using a PELICAN_CLASS instance and its run method.
+
-    """
+    _SITE_DB.keys() need to be ready for filter_translations
-    global _main_site_generated
+    '''
-    if _main_site_generated:      # make sure this is only called once
+    _SITE_DB.clear()
-        return
+    _SITE_DB[_MAIN_LANG] = _MAIN_SITEURL
+    # make sure it works for both root-relative and absolute
+    main_siteurl = '/' if _MAIN_SITEURL == '' else _MAIN_SITEURL
+    for lang, overrides in _SUBSITE_QUEUE.items():
+        if 'SITEURL' not in overrides:
+            overrides['SITEURL'] = posixpath.join(main_siteurl, lang)
+        _SITE_DB[lang] = overrides['SITEURL']
+        # default subsite hierarchy
+        if 'OUTPUT_PATH' not in overrides:
+            overrides['OUTPUT_PATH'] = os.path.join(
+                _MAIN_SETTINGS['OUTPUT_PATH'], lang)
+        if 'CACHE_PATH' not in overrides:
+            overrides['CACHE_PATH'] = os.path.join(
+                _MAIN_SETTINGS['CACHE_PATH'], lang)
+        if 'STATIC_PATHS' not in overrides:
+            overrides['STATIC_PATHS'] = []
+        if ('THEME' not in overrides and 'THEME_STATIC_DIR' not in overrides and
+                'THEME_STATIC_PATHS' not in overrides):
+            relpath = relpath_to_site(lang, _MAIN_LANG)
+            overrides['THEME_STATIC_DIR'] = posixpath.join(
+                relpath, _MAIN_SETTINGS['THEME_STATIC_DIR'])
+            overrides['THEME_STATIC_PATHS'] = []
+        # to change what is perceived as translations
+        overrides['DEFAULT_LANG'] = lang
+
+
+def subscribe_filter_to_signals(settings):
+    '''Subscribe content filter to requested signals'''
+    for sig in settings.get('I18N_FILTER_SIGNALS', []):
+        sig.connect(filter_contents_translations)
+
+
+def initialize_plugin(pelican_obj):
+    '''Initialize plugin variables and Pelican settings'''
+    if _MAIN_SETTINGS is None:
+        initialize_dbs(pelican_obj.settings)
+        subscribe_filter_to_signals(pelican_obj.settings)
+
+
+def get_site_path(url):
+    '''Get the path component of an url, excludes siteurl
+
+    also normalizes '' to '/' for relpath to work,
+    otherwise it could be interpreted as a relative filesystem path
+    '''
+    path = urlparse(url).path
+    if path == '':
+        path = '/'
+    return path
+
+
+def relpath_to_site(lang, target_lang):
+    '''Get relative path from siteurl of lang to siteurl of base_lang
+
+    the output is cached in _SITES_RELPATH_DB
+    '''
+    path = _SITES_RELPATH_DB.get((lang, target_lang), None)
+    if path is None:
+        siteurl = _SITE_DB.get(lang, _MAIN_SITEURL)
+        target_siteurl = _SITE_DB.get(target_lang, _MAIN_SITEURL)
+        path = posixpath.relpath(get_site_path(target_siteurl),
+                                 get_site_path(siteurl))
+        _SITES_RELPATH_DB[(lang, target_lang)] = path
+    return path
+
+
+def save_generator(generator):
+    '''Save the generator for later use
+
+    initialize the removed content list
+    '''
+    _GENERATOR_DB[generator] = []
+
+
+def article2draft(article):
+    '''Transform an Article to Draft'''
+    draft = Draft(article._content, article.metadata, article.settings,
+                  article.source_path, article._context)
+    draft.status = 'draft'
+    return draft
+
+
+def page2hidden_page(page):
+    '''Transform a Page to a hidden Page'''
+    page.status = 'hidden'
+    return page
+
+
+class GeneratorInspector(object):
+    '''Inspector of generator instances'''
+
+    generators_info = {
+        ArticlesGenerator: {
+            'translations_lists': ['translations', 'drafts_translations'],
+            'contents_lists': [('articles', 'drafts')],
+            'hiding_func': article2draft,
+            'policy': 'I18N_UNTRANSLATED_ARTICLES',
+        },
+        PagesGenerator: {
+            'translations_lists': ['translations', 'hidden_translations'],
+            'contents_lists': [('pages', 'hidden_pages')],
+            'hiding_func': page2hidden_page,
+            'policy': 'I18N_UNTRANSLATED_PAGES',
+        },
+    }
+
+    def __init__(self, generator):
+        '''Identify the best known class of the generator instance
+
+        The class '''
+        self.generator = generator
+        self.generators_info.update(generator.settings.get(
+            'I18N_GENERATORS_INFO', {}))
+        for cls in generator.__class__.__mro__:
+            if cls in self.generators_info:
+                self.info = self.generators_info[cls]
+                break
         else:
-        _main_site_generated = True
+            self.info = {}
 
-    orig_settings = pelican_obj.settings
+    def translations_lists(self):
-    for lang, overrides in orig_settings.get('I18N_SUBSITES', {}).items():
+        '''Iterator over lists of content translations'''
-        settings = orig_settings.copy()
+        return (getattr(self.generator, name) for name in
-        settings.update(overrides)
+                self.info.get('translations_lists', []))
-        settings['SITEURL'] = _lang_siteurls[lang]
-        settings['OUTPUT_PATH'] = os.path.join(orig_settings['OUTPUT_PATH'], lang, '')
-        settings['DEFAULT_LANG'] = lang   # to change what is perceived as translations
-        settings['DELETE_OUTPUT_DIRECTORY'] = False  # prevent deletion of previous runs
-        settings = configure_settings(settings)      # to set LOCALE, etc.
 
-        cls = settings['PELICAN_CLASS']
+    def contents_list_pairs(self):
-        if isinstance(cls, six.string_types):
+        '''Iterator over pairs of normal and hidden contents'''
-            module, cls_name = cls.rsplit('.', 1)
+        return (tuple(getattr(self.generator, name) for name in names)
-            module = __import__(module)
+                for names in self.info.get('contents_lists', []))
-            cls = getattr(module, cls_name)
 
-        pelican_obj = cls(settings)
+    def hiding_function(self):
-        logger.debug("Generating i18n subsite for lang '{}' using class '{}'".format(lang, str(cls)))
+        '''Function for transforming content to a hidden version'''
-        pelican_obj.run()
+        hiding_func = self.info.get('hiding_func', lambda x: x)
-    _main_site_generated = False          # for autoreload mode
+        return hiding_func
+
+    def untranslated_policy(self, default):
+        '''Get the policy for untranslated content'''
+        return self.generator.settings.get(self.info.get('policy', None),
+                                           default)
+
+    def all_contents(self):
+        '''Iterator over all contents'''
+        translations_iterator = chain(*self.translations_lists())
+        return chain(translations_iterator,
+                     *(pair[i] for pair in self.contents_list_pairs()
+                       for i in (0, 1)))
 
 
+def filter_contents_translations(generator):
+    '''Filter the content and translations lists of a generator
 
-def move_translations_links(content_object):
+    Filters out
-    """This function points translations links to the sub-sites
+        1) translations which will be generated in a different site
+        2) content that is not in the language of the currently
+        generated site but in that of a different site, content in a
+        language which has no site is generated always. The filtering
+        method bay be modified by the respective untranslated policy
+    '''
+    inspector = GeneratorInspector(generator)
+    current_lang = generator.settings['DEFAULT_LANG']
+    langs_with_sites = _SITE_DB.keys()
+    removed_contents = _GENERATOR_DB[generator]
 
-    by prepending their location with the language code
+    for translations in inspector.translations_lists():
-    or directs an original DEFAULT_LANG translation back to top level site
+        for translation in translations[:]:    # copy to be able to remove
-    """
+            if translation.lang in langs_with_sites:
-    for translation in content_object.translations:
+                translations.remove(translation)
-        if translation.lang == _main_site_lang:
+                removed_contents.append(translation)
-        # cannot prepend, must take to top level
-            lang_prepend = '../'
-        else:
-            lang_prepend = translation.lang + '/'
-        translation.override_url =  lang_prepend + translation.url
-
-
-
-def update_generator_contents(generator, *args):
-    """Update the contents lists of a generator
-
-    Empty the (hidden_)translation attribute of article and pages generators
-    to prevent generating the translations as they will be generated in the lang sub-site
-    and point the content translations links to the sub-sites
-
-    Hide content without a translation for current DEFAULT_LANG
-    if HIDE_UNTRANSLATED_CONTENT is True
-    """
-    generator.translations = []
-    is_pages_gen = hasattr(generator, 'pages')
-    if is_pages_gen:
-        generator.hidden_translations = []
-        for page in chain(generator.pages, generator.hidden_pages):
-            move_translations_links(page)
-    else:                                    # is an article generator
-        for article in chain(generator.articles, generator.drafts):
-            move_translations_links(article)
-
-    if not generator.settings.get('HIDE_UNTRANSLATED_CONTENT', True):
-        return
-    contents = generator.pages if is_pages_gen else generator.articles
-    hidden_contents = generator.hidden_pages if is_pages_gen else generator.drafts
-    default_lang = generator.settings['DEFAULT_LANG']
-    for content_object in contents[:]:   # loop over copy for removing
-        if content_object.lang != default_lang:
-            if isinstance(content_object, Article):
-                content_object.status = 'draft'
-            elif isinstance(content_object, Page):
-                content_object.status = 'hidden'
-            contents.remove(content_object)
-            hidden_contents.append(content_object)
-    if not is_pages_gen: # regenerate categories, tags, etc. for articles
-        if hasattr(generator, '_generate_context_aggregate'):                  # if implemented
-            # Simulate __init__ for fields that need it
-            generator.dates = {}
-            generator.tags = defaultdict(list)
-            generator.categories = defaultdict(list)
-            generator.authors = defaultdict(list)
-            generator._generate_context_aggregate()
-        else:                             # fallback for Pelican 3.3.0
-            regenerate_context_articles(generator)
 
+    hiding_func = inspector.hiding_function()
+    untrans_policy = inspector.untranslated_policy(default='hide')
+    for (contents, other_contents) in inspector.contents_list_pairs():
+        for content in contents[:]:        # copy for removing in loop
+            if content.lang == current_lang: # in native lang
+                # save the native URL attr formatted in the current locale
+                _NATIVE_CONTENT_URL_DB[content.source_path] = content.url
+            elif content.lang in langs_with_sites and untrans_policy != 'keep':
+                contents.remove(content)
+                if untrans_policy == 'hide':
+                    other_contents.append(hiding_func(content))
+                elif untrans_policy == 'remove':
+                    removed_contents.append(content)
 
 
 def install_templates_translations(generator):
-    """Install gettext translations for current DEFAULT_LANG in the jinja2.Environment
+    '''Install gettext translations in the jinja2.Environment
 
-    if the 'jinja2.ext.i18n' jinja2 extension is enabled
+    Only if the 'jinja2.ext.i18n' jinja2 extension is enabled
-    adds some useful variables into the template context
+    the translations for the current DEFAULT_LANG are installed.
-    """
+    '''
-    generator.context['main_siteurl'] = _main_siteurl
+    if 'jinja2.ext.i18n' in generator.settings['JINJA_EXTENSIONS']:
-    generator.context['main_lang'] = _main_site_lang
-    generator.context['lang_siteurls'] = _lang_siteurls
-    current_def_lang = generator.settings['DEFAULT_LANG']
-    extra_siteurls = _lang_siteurls.copy()
-    extra_siteurls.pop(current_def_lang)
-    generator.context['extra_siteurls'] = extra_siteurls
-    
-    if 'jinja2.ext.i18n' not in generator.settings['JINJA_EXTENSIONS']:
-        return
         domain = generator.settings.get('I18N_GETTEXT_DOMAIN', 'messages')
         localedir = generator.settings.get('I18N_GETTEXT_LOCALEDIR')
         if localedir is None:
             localedir = os.path.join(generator.theme, 'translations')
-    if current_def_lang == generator.settings.get('I18N_TEMPLATES_LANG', _main_site_lang):
+        current_lang = generator.settings['DEFAULT_LANG']
+        if current_lang == generator.settings.get('I18N_TEMPLATES_LANG',
+                                                  _MAIN_LANG):
             translations = gettext.NullTranslations()
         else:
-        languages = [current_def_lang]
+            langs = [current_lang]
             try:
-            translations = gettext.translation(domain, localedir, languages)
+                translations = gettext.translation(domain, localedir, langs)
             except (IOError, OSError):
-            logger.error("Cannot find translations for language '{}' in '{}' with domain '{}'. Installing NullTranslations.".format(languages[0], localedir, domain))
+                _LOGGER.error((
+                    "Cannot find translations for language '{}' in '{}' with "
+                    "domain '{}'. Installing NullTranslations.").format(
+                        langs[0], localedir, domain))
                 translations = gettext.NullTranslations()
         newstyle = generator.settings.get('I18N_GETTEXT_NEWSTYLE', True)
         generator.env.install_gettext_translations(translations, newstyle)
 
 
+def add_variables_to_context(generator):
+    '''Adds useful iterable variables to template context'''
+    context = generator.context             # minimize attr lookup
+    context['relpath_to_site'] = relpath_to_site
+    context['main_siteurl'] = _MAIN_SITEURL
+    context['main_lang'] = _MAIN_LANG
+    context['lang_siteurls'] = _SITE_DB
+    current_lang = generator.settings['DEFAULT_LANG']
+    extra_siteurls = _SITE_DB.copy()
+    extra_siteurls.pop(current_lang)
+    context['extra_siteurls'] = extra_siteurls
+
+
+def interlink_translations(content):
+    '''Link content to translations in their main language
+
+    so the URL (including localized month names) of the different subsites
+    will be honored
+    '''
+    lang = content.lang
+    # sort translations by lang
+    content.translations.sort(key=attrgetter('lang'))
+    for translation in content.translations:
+        relpath = relpath_to_site(lang, translation.lang)
+        url = _NATIVE_CONTENT_URL_DB[translation.source_path]
+        translation.override_url = posixpath.join(relpath, url)
+
+
+def interlink_translated_content(generator):
+    '''Make translations link to the native locations
+
+    for generators that may contain translated content
+    '''
+    inspector = GeneratorInspector(generator)
+    for content in inspector.all_contents():
+        interlink_translations(content)
+
+
+def interlink_removed_content(generator):
+    '''For all contents removed from generation queue update interlinks
+
+    link to the native location
+    '''
+    current_lang = generator.settings['DEFAULT_LANG']
+    for content in _GENERATOR_DB[generator]:
+        url = _NATIVE_CONTENT_URL_DB[content.source_path]
+        relpath = relpath_to_site(current_lang, content.lang)
+        content.override_url = posixpath.join(relpath, url)
+
+
+def interlink_static_files(generator):
+    '''Add links to static files in the main site if necessary'''
+    if generator.settings['STATIC_PATHS'] != []:
+        return                               # customized STATIC_PATHS
+    filenames = generator.context['filenames'] # minimize attr lookup
+    relpath = relpath_to_site(generator.settings['DEFAULT_LANG'], _MAIN_LANG)
+    for staticfile in _MAIN_STATIC_FILES:
+        if staticfile.get_relative_source_path() not in filenames:
+            staticfile = copy(staticfile) # prevent override in main site
+            staticfile.override_url = posixpath.join(relpath, staticfile.url)
+            generator.add_source_path(staticfile)
+
+
+def save_main_static_files(static_generator):
+    '''Save the static files generated for the main site'''
+    global _MAIN_STATIC_FILES
+    # test just for current lang as settings change in autoreload mode
+    if static_generator.settings['DEFAULT_LANG'] == _MAIN_LANG:
+        _MAIN_STATIC_FILES = static_generator.staticfiles
+
+
+def update_generators():
+    '''Update the context of all generators
+
+    Ads useful variables and translations into the template context
+    and interlink translations
+    '''
+    for generator in _GENERATOR_DB.keys():
+        install_templates_translations(generator)
+        add_variables_to_context(generator)
+        interlink_static_files(generator)
+        interlink_removed_content(generator)
+        interlink_translated_content(generator)
+
+
+def get_pelican_cls(settings):
+    '''Get the Pelican class requested in settings'''
+    cls = settings['PELICAN_CLASS']
+    if isinstance(cls, six.string_types):
+        module, cls_name = cls.rsplit('.', 1)
+        module = __import__(module)
+        cls = getattr(module, cls_name)
+    return cls
+
+
+def create_next_subsite(pelican_obj):
+    '''Create the next subsite using the lang-specific config
+
+    If there are no more subsites in the generation queue, update all
+    the generators (interlink translations and removed content, add
+    variables and translations to template context). Otherwise get the
+    language and overrides for next the subsite in the queue and apply
+    overrides.  Then generate the subsite using a PELICAN_CLASS
+    instance and its run method. Finally, restore the previous locale.
+    '''
+    global _MAIN_SETTINGS
+    if len(_SUBSITE_QUEUE) == 0:
+        _LOGGER.debug(
+            'i18n: Updating cross-site links and context of all generators.')
+        update_generators()
+        _MAIN_SETTINGS = None             # to initialize next time
+    else:
+        with temporary_locale():
+            settings = _MAIN_SETTINGS.copy()
+            lang, overrides = _SUBSITE_QUEUE.popitem()
+            settings.update(overrides)
+            settings = configure_settings(settings)      # to set LOCALE, etc.
+            cls = get_pelican_cls(settings)
+
+            new_pelican_obj = cls(settings)
+            _LOGGER.debug(("Generating i18n subsite for language '{}' "
+                           "using class {}").format(lang, cls))
+            new_pelican_obj.run()
+
+
+# map: signal name -> function name
+_SIGNAL_HANDLERS_DB = {
+    'get_generators': initialize_plugin,
+    'article_generator_pretaxonomy': filter_contents_translations,
+    'page_generator_finalized': filter_contents_translations,
+    'get_writer': create_next_subsite,
+    'static_generator_finalized': save_main_static_files,
+    'generator_init': save_generator,
+}
+
 
 def register():
-    signals.initialized.connect(disable_lang_vars)
+    '''Register the plugin only if required signals are available'''
-    signals.generator_init.connect(install_templates_translations)
+    for sig_name in _SIGNAL_HANDLERS_DB.keys():
-    signals.article_generator_finalized.connect(update_generator_contents)
+        if not hasattr(signals, sig_name):
-    signals.page_generator_finalized.connect(update_generator_contents)
+            _LOGGER.error((
-    signals.finalized.connect(create_lang_subsites)
+                'The i18n_subsites plugin requires the {} '
+                'signal available for sure in Pelican 3.4.0 and later, '
+                'plugin will not be used.').format(sig_name))
+            return
+
+    for sig_name, handler in _SIGNAL_HANDLERS_DB.items():
+        sig = getattr(signals, sig_name)
+        sig.connect(handler)

implementing_language_buttons.rst

@@ -2,9 +2,12 @@
 Implementing language buttons
 -----------------------------
 
-Each article with translations has translations links, but that's the only way to switch between language subsites.
+Each article with translations has translations links, but that's the
+only way to switch between language subsites.
 
-For this reason it is convenient to add language buttons to the top menu bar to make it simple to switch between the language subsites on all pages.
+For this reason it is convenient to add language buttons to the top
+menu bar to make it simple to switch between the language subsites on
+all pages.
 
 Example designs
 ---------------
@@ -12,7 +15,9 @@ Example designs
 Language buttons showing other available languages
 ..................................................
 
-The ``extra_siteurls`` dictionary is a mapping of all other (not the *DEFAULT_LANG* of the current (sub-)site) languages to the *SITEURL* of the respective (sub-)sites
+The ``extra_siteurls`` dictionary is a mapping of all other (not the
+``DEFAULT_LANG`` of the current (sub-)site) languages to the
+``SITEURL`` of the respective (sub-)sites
 
 .. code-block:: jinja
 
@@ -20,7 +25,7 @@ The ``extra_siteurls`` dictionary is a mapping of all other (not the *DEFAULT_LA
    <nav><ul>
    {% if extra_siteurls %}
    {% for lang, url in extra_siteurls.items() %}
-   <li><a href="{{ url }}">{{ lang }}</a></li>
+   <li><a href="{{ url }}/">{{ lang }}</a></li>
    {% endfor %}
    <!-- separator -->
    <li style="background-color: white; padding: 5px;">&nbsp</li>
@@ -28,10 +33,15 @@ The ``extra_siteurls`` dictionary is a mapping of all other (not the *DEFAULT_LA
    {% for title, link in MENUITEMS %}
    <!-- SNIP -->
 
-Language buttons showing all available languages, current is active
+Notice the slash ``/`` after ``{{ url }}``, this makes sure it works
-..................................................................
+with local development when ``SITEURL == ''``.
 
-The ``extra_siteurls`` dictionary is a mapping of all languages to the *SITEURL* of the respective (sub-)sites. This template sets the language of the current (sub-)site as active.
+Language buttons showing all available languages, current is active
+...................................................................
+
+The ``extra_siteurls`` dictionary is a mapping of all languages to the
+``SITEURL`` of the respective (sub-)sites. This template sets the
+language of the current (sub-)site as active.
 
 .. code-block:: jinja
 
@@ -39,7 +49,7 @@ The ``extra_siteurls`` dictionary is a mapping of all languages to the *SITEURL*
    <nav><ul>
    {% if lang_siteurls %}
    {% for lang, url in lang_siteurls.items() %}
-   <li{% if lang == DEFAULT_LANG %} class="active"{% endif %}><a href="{{ url }}">{{ lang }}</a></li>
+   <li{% if lang == DEFAULT_LANG %} class="active"{% endif %}><a href="{{ url }}/">{{ lang }}</a></li>
    {% endfor %}
    <!-- separator -->
    <li style="background-color: white; padding: 5px;">&nbsp</li>
@@ -54,7 +64,9 @@ Tips and tricks
 Showing more than language codes
 ................................
 
-If you want the language buttons to show e.g. the names of the languages or flags [#flags]_, not just the language codes, you can use a jinja filter to translate the language codes
+If you want the language buttons to show e.g. the names of the
+languages or flags [#flags]_, not just the language codes, you can use
+a jinja filter to translate the language codes
 
 
 .. code-block:: python
@@ -78,7 +90,7 @@ And then the link content becomes
 
    <!-- SNIP -->
    {% for lang, siteurl in lang_siteurls.items() %}
-   <li{% if lang == DEFAULT_LANG %} class="active"{% endif %}><a href="{{ siteurl }}">{{ lang | lookup_lang_name }}</a></li>
+   <li{% if lang == DEFAULT_LANG %} class="active"{% endif %}><a href="{{ siteurl }}/">{{ lang | lookup_lang_name }}</a></li>
    {% endfor %}
    <!-- SNIP -->
 
@@ -86,7 +98,9 @@ And then the link content becomes
 Changing the order of language buttons
 ......................................
 
-Because ``lang_siteurls`` and ``extra_siteurls`` are instances of ``OrderedDict`` with ``main_lang`` being always the first key, you can change the order through a jinja filter.
+Because ``lang_siteurls`` and ``extra_siteurls`` are instances of
+``OrderedDict`` with ``main_lang`` being always the first key, you can
+change the order through a jinja filter.
 
 .. code-block:: python
 
@@ -110,4 +124,5 @@ And then the ``for`` loop line in the template becomes
    <!-- SNIP -->
 
 
-.. [#flags] Although it may look nice, `w3 discourages it <http://www.w3.org/TR/i18n-html-tech-lang/#ri20040808.173208643>`_.
+.. [#flags] Although it may look nice, `w3 discourages it
+            <http://www.w3.org/TR/i18n-html-tech-lang/#ri20040808.173208643>`_.

localizing_using_jinja2.rst

@@ -6,33 +6,46 @@ Localizing themes with Jinja2
 ---------------------
 
 To enable the |ext| extension in your templates, you must add it to
-*JINJA_EXTENSIONS* in your Pelican configuration
+``JINJA_EXTENSIONS`` in your Pelican configuration
 
 .. code-block:: python
 
   JINJA_EXTENSIONS = ['jinja2.ext.i18n', ...]
 
-Then follow the `Jinja2 templating documentation for the I18N plugin <http://jinja.pocoo.org/docs/templates/#i18n>`_ to make your templates localizable. This usually means surrounding strings with the ``{% trans %}`` directive or using ``gettext()`` in expressions
+Then follow the `Jinja2 templating documentation for the I18N plugin
+<http://jinja.pocoo.org/docs/templates/#i18n>`_ to make your templates
+localizable. This usually means surrounding strings with the ``{%
+trans %}`` directive or using ``gettext()`` in expressions
 
 .. code-block:: jinja
 
     {% trans %}translatable content{% endtrans %}
     {{ gettext('a translatable string') }}
 
-For pluralization support, etc. consult the documentation
+For pluralization support, etc. consult the documentation.
 
-To enable `newstyle gettext calls <http://jinja.pocoo.org/docs/extensions/#newstyle-gettext>`_ the *I18N_GETTEXT_NEWSTYLE* config variable must be set to ``True`` (default).
+To enable `newstyle gettext calls
+<http://jinja.pocoo.org/docs/extensions/#newstyle-gettext>`_ the
+``I18N_GETTEXT_NEWSTYLE`` config variable must be set to ``True``
+(default).
 
 .. |ext| replace:: ``jinja2.ext.i18n``
 
 2. Specify translations location
 --------------------------------
 
-The |ext| extension uses the `Python gettext library <http://docs.python.org/library/gettext.html>`_ for translating strings.
+The |ext| extension uses the `Python gettext library
+<http://docs.python.org/library/gettext.html>`_ for translating
+strings.
 
-In your Pelican config you can give the path in which to look for translations in the *I18N_GETTEXT_LOCALEDIR* variable. If not given, it is assumed to be the ``translations`` subfolder in the top folder of the theme specified by *THEME*.
+In your Pelican config you can give the path in which to look for
+translations in the ``I18N_GETTEXT_LOCALEDIR`` variable. If not given,
+it is assumed to be the ``translations`` subfolder in the top folder
+of the theme specified by ``THEME``.
 
-The domain of the translations (the name of each translation file is ``domain.mo``) is controlled by the *I18N_GETTEXT_DOMAIN* config variable (defaults to ``messages``).
+The domain of the translations (the name of each translation file is
+``domain.mo``) is controlled by the ``I18N_GETTEXT_DOMAIN`` config
+variable (defaults to ``messages``).
 
 Example
 .......
@@ -44,39 +57,61 @@ With the following in your Pelican settings file
   I18N_GETTEXT_LOCALEDIR = 'some/path/'
   I18N_GETTEXT_DOMAIN = 'my_domain'
 
-… the translation for language 'cz' will be expected to be in ``some/path/cz/LC_MESSAGES/my_domain.mo``
+the translation for language 'cz' will be expected to be in
+``some/path/cz/LC_MESSAGES/my_domain.mo``
 
 
 3. Extract translatable strings and translate them
 --------------------------------------------------
 
-There are many ways to extract translatable strings and create ``gettext`` compatible translations. You can create the ``*.po`` and ``*.mo`` message catalog files yourself, or you can use some helper tool as described in `the Python gettext library tutorial <http://docs.python.org/library/gettext.html#internationalizing-your-programs-and-modules>`_.
+There are many ways to extract translatable strings and create
+``gettext`` compatible translations. You can create the ``*.po`` and
+``*.mo`` message catalog files yourself, or you can use some helper
+tool as described in `the Python gettext library tutorial
+<http://docs.python.org/library/gettext.html#internationalizing-your-programs-and-modules>`_.
 
-You of course don't need to provide a translation for the language in which the templates are written which is assumed to be the original *DEFAULT_LANG*. This can be overridden in the *I18N_TEMPLATES_LANG* variable.
+You of course don't need to provide a translation for the language in
+which the templates are written which is assumed to be the original
+``DEFAULT_LANG``. This can be overridden in the
+``I18N_TEMPLATES_LANG`` variable.
 
 Recommended tool: babel
 .......................
 
-`Babel <http://babel.pocoo.org/>`_ makes it easy to extract translatable strings from the localized Jinja2 templates and assists with creating translations as documented in this `Jinja2-Babel tutorial <http://pythonhosted.org/Flask-Babel/#translating-applications>`_ [#flask]_ on which the following is based.
+`Babel <http://babel.pocoo.org/>`_ makes it easy to extract
+translatable strings from the localized Jinja2 templates and assists
+with creating translations as documented in this `Jinja2-Babel
+tutorial
+<http://pythonhosted.org/Flask-Babel/#translating-applications>`_
+[#flask]_ on which the following is based.
 
 1. Add babel mapping
 ~~~~~~~~~~~~~~~~~~~~
 
-Let's assume that you are localizing a theme in ``themes/my_theme/`` and that you use the default settings, i.e. the default domain ``messages`` and will put the translations in the ``translations`` subdirectory of the theme directory as ``themes/my_theme/translations/``.
+Let's assume that you are localizing a theme in ``themes/my_theme/``
+and that you use the default settings, i.e. the default domain
+``messages`` and will put the translations in the ``translations``
+subdirectory of the theme directory as
+``themes/my_theme/translations/``.
 
-It is up to you where to store babel mappings and translation files templates (``*.pot``), but a convenient place is to put them in ``themes/my_theme/`` and work in that directory. From now on let's assume that it will be our current working directory (CWD).
+It is up to you where to store babel mappings and translation files
+templates (``*.pot``), but a convenient place is to put them in
+``themes/my_theme/`` and work in that directory. From now on let's
+assume that it will be our current working directory (CWD).
 
-To tell babel to extract translatable strings from the templates create a mapping file ``babel.cfg`` with the following line
+To tell babel to extract translatable strings from the templates
+create a mapping file ``babel.cfg`` with the following line
 
 .. code-block:: cfg
 
-    [jinja2: ./templates/**.html]
+    [jinja2: templates/**.html]
 
 
 2. Extract translatable strings from templates
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Run the following command to create a ``messages.pot`` message catalog template file from extracted translatable strings
+Run the following command to create a ``messages.pot`` message catalog
+template file from extracted translatable strings
 
 .. code-block:: bash
 
@@ -86,19 +121,28 @@ Run the following command to create a ``messages.pot`` message catalog template
 3. Initialize message catalogs
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you want to translate the template to language ``lang``, run the following command to create a message catalog
+If you want to translate the template to language ``lang``, run the
-``translations/lang/LC_MESSAGES/messages.po`` using the template ``messages.pot``
+following command to create a message catalog
+``translations/lang/LC_MESSAGES/messages.po`` using the template
+``messages.pot``
 
 .. code-block:: bash
 
     pybabel init --input-file messages.pot --output-dir translations/ --locale lang --domain messages
 
-babel expects ``lang`` to be a valid locale identifier, so if e.g. you are translating for language ``cz`` but the corresponding locale is ``cs``, you have to use the locale identifier. Nevertheless, the gettext infrastructure should later correctly find the locale for a given language.
+babel expects ``lang`` to be a valid locale identifier, so if e.g. you
+are translating for language ``cz`` but the corresponding locale is
+``cs``, you have to use the locale identifier. Nevertheless, the
+gettext infrastructure should later correctly find the locale for a
+given language.
 
 4. Fill the message catalogs
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The message catalog files format is quite intuitive, it is fully documented in the `GNU gettext manual <http://www.gnu.org/software/gettext/manual/gettext.html#PO-Files>`_. Essentially, you fill in the ``msgstr`` strings
+The message catalog files format is quite intuitive, it is fully
+documented in the `GNU gettext manual
+<http://www.gnu.org/software/gettext/manual/gettext.html#PO-Files>`_. Essentially,
+you fill in the ``msgstr`` strings
 
 
 .. code-block:: po
@@ -113,30 +157,44 @@ The message catalog files format is quite intuitive, it is fully documented in t
     "nějaký více řádkový řetězec"
     "vypadá takto"
 
-You might also want to remove ``#,fuzzy`` flags once the translation is complete and reviewed to show that it can be compiled.
+You might also want to remove ``#,fuzzy`` flags once the translation
+is complete and reviewed to show that it can be compiled.
 
 5. Compile the message catalogs
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The message catalogs must be compiled into binary format using this command
+The message catalogs must be compiled into binary format using this
+command
 
 .. code-block:: bash
 
     pybabel compile --directory translations/ --domain messages
 
-This command might complain about "fuzzy" translations, which means you should review the translations and once done, remove the fuzzy flag line.
+This command might complain about "fuzzy" translations, which means
+you should review the translations and once done, remove the fuzzy
+flag line.
 
 (6.) Update the catalogs when templates change
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you add any translatable patterns into your templates, you have to update your message catalogs too.
+If you add any translatable patterns into your templates, you have to
-First you extract a new message catalog template as described in the 2. step. Then you run the following command [#pybabel_error]_
+update your message catalogs too.  First you extract a new message
+catalog template as described in the 2. step. Then you run the
+following command [#pybabel_error]_
 
 .. code-block:: bash
 
    pybabel update --input-file messages.pot --output-dir translations/ --domain messages
 
-This will merge the new patterns with the old ones. Once you review and fill them, you have to recompile them as described in the 5. step.
+This will merge the new patterns with the old ones. Once you review
+and fill them, you have to recompile them as described in the 5. step.
 
-.. [#flask] Although the tutorial is focused on Flask-based web applications, the linked translation tutorial is not Flask-specific.
+.. [#flask] Although the tutorial is focused on Flask-based web
-.. [#pybabel_error] If you get an error ``TypeError: must be str, not bytes`` with Python 3.3, it is likely you are suffering from this `bug <https://github.com/mitsuhiko/flask-babel/issues/43>`_. Until the fix is released, you can use babel with Python 2.7.
+            applications, the linked translation tutorial is not
+            Flask-specific.
+.. [#pybabel_error] If you get an error ``TypeError: must be str, not
+                    bytes`` with Python 3.3, it is likely you are
+                    suffering from this `bug
+                    <https://github.com/mitsuhiko/flask-babel/issues/43>`_.
+                    Until the fix is released, you can use babel with
+                    Python 2.7.

test_data/content/pages/untranslated-page.rst

@@ -0,0 +1,5 @@
+Untranslated page
+=================
+:lang: en
+
+This page has no translation.

test_data/content/translated_article-cz.rst

@@ -0,0 +1,8 @@
+Přeložený článek
+================
+:slug: translated-article
+:lang: cz
+:date: 2014-09-15
+
+Jednoduchý článek s překlady.
+Zde je odkaz na `nějaký obrázek <{filename}/images/img.png>`_.

test_data/content/translated_article-de.rst

@@ -0,0 +1,8 @@
+Ein übersetzter Artikel
+=======================
+:slug: translated-article
+:lang: de
+:date: 2014-09-14
+
+Ein einfacher Artikel mit einer Übersetzung.
+Hier ist ein Link zur `einigem Bild <{filename}/images/img.png>`_.

test_data/content/translated_article-en.rst

@@ -0,0 +1,8 @@
+A translated article
+====================
+:slug: translated-article
+:lang: en
+:date: 2014-09-13
+
+A simple article with a translation.
+Here is a link to `some image <{filename}/images/img.png>`_.

test_data/content/untranslated_article-en.rst

@@ -0,0 +1,9 @@
+An untranslated article
+=======================
+:date: 2014-07-14
+:lang: en
+
+An article without a translation.
+Here is a link to an `untranslated page`_
+
+.. _`untranslated page`: {filename}/pages/untranslated-page.rst

test_data/localized_theme/babel.cfg

@@ -0,0 +1,2 @@
+[jinja2: templates/**.html]
+

test_data/localized_theme/messages.pot

@@ -0,0 +1,23 @@
+# Translations template for PROJECT.
+# Copyright (C) 2014 ORGANIZATION
+# This file is distributed under the same license as the PROJECT project.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2014.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: PROJECT VERSION\n"
+"Report-Msgid-Bugs-To: EMAIL@ADDRESS\n"
+"POT-Creation-Date: 2014-07-13 12:25+0200\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 1.3\n"
+
+#: templates/base.html:3
+msgid "Welcome to our"
+msgstr ""
+

test_data/localized_theme/templates/base.html

@@ -0,0 +1,7 @@
+{% extends "!simple/base.html" %}
+
+{% block title %}{% trans %}Welcome to our{% endtrans %} {{ SITENAME }}{% endblock %}
+{% block head %}
+{{ super() }}
+<link rel="stylesheet" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/style.css" />
+{% endblock %}

test_data/localized_theme/translations/de/LC_MESSAGES/messages.po

@@ -0,0 +1,23 @@
+# German translations for PROJECT.
+# Copyright (C) 2014 ORGANIZATION
+# This file is distributed under the same license as the PROJECT project.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2014.
+#
+msgid ""
+msgstr ""
+"Project-Id-Version: PROJECT VERSION\n"
+"Report-Msgid-Bugs-To: EMAIL@ADDRESS\n"
+"POT-Creation-Date: 2014-07-13 12:25+0200\n"
+"PO-Revision-Date: 2014-07-13 12:26+0200\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: de <LL@li.org>\n"
+"Plural-Forms: nplurals=2; plural=(n != 1)\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 1.3\n"
+
+#: templates/base.html:3
+msgid "Welcome to our"
+msgstr "Willkommen Sie zur unserer"
+

test_data/output/an-untranslated-article.html

@@ -0,0 +1,50 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+          <title>Welcome to our Testing site</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testing site Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/theme/style.css" />
+
+
+
+
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/">Testing site <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+            <li><a href="http://example.com/test/pages/untranslated-page.html">Untranslated page</a></li>
+        </ul></nav><!-- /#menu -->
+<section id="content" class="body">
+  <header>
+    <h2 class="entry-title">
+      <a href="http://example.com/test/an-untranslated-article.html" rel="bookmark"
+         title="Permalink to An untranslated article">An untranslated article</a></h2>
+ 
+  </header>
+  <footer class="post-info">
+    <abbr class="published" title="2014-07-14T00:00:00">
+      Mon 14 July 2014
+    </abbr>
+    <address class="vcard author">
+      By           <a class="url fn" href="http://example.com/test/author/the-tester.html">The Tester</a>
+    </address>
+  </footer><!-- /.post-info -->
+  <div class="entry-content">
+    <p>An article without a translation.
+Here is a link to an <a class="reference external" href="http://example.com/test/pages/untranslated-page.html">untranslated page</a></p>
+
+  </div><!-- /.entry-content -->
+</section>
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/cz/an-untranslated-article-en.html

@@ -0,0 +1,49 @@
+<!DOCTYPE html>
+<html lang="cz">
+<head>
+          <title>Welcome to our Testovací stránka</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testovací stránka Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/cz/../theme/style.css" />
+
+
+
+
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/cz/">Testovací stránka <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+        </ul></nav><!-- /#menu -->
+<section id="content" class="body">
+  <header>
+    <h2 class="entry-title">
+      <a href="http://example.com/test/cz/an-untranslated-article-en.html" rel="bookmark"
+         title="Permalink to An untranslated article">An untranslated article</a></h2>
+ 
+  </header>
+  <footer class="post-info">
+    <abbr class="published" title="2014-07-14T00:00:00">
+      Mon 14 July 2014
+    </abbr>
+    <address class="vcard author">
+      By           <a class="url fn" href="http://example.com/test/cz/author/test-testovic.html">Test Testovič</a>
+    </address>
+  </footer><!-- /.post-info -->
+  <div class="entry-content">
+    <p>An article without a translation.
+Here is a link to an <a class="reference external" href="http://example.com/test/cz/../pages/untranslated-page.html">untranslated page</a></p>
+
+  </div><!-- /.entry-content -->
+</section>
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/cz/feeds_all.atom.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Testovací stránka</title><link href="http://example.com/test/cz/" rel="alternate"></link><link href="http://example.com/test/feeds_all.atom.xml" rel="self"></link><id>http://example.com/test/cz/</id><updated>2014-09-15T00:00:00+00:00</updated><entry><title>Přeložený článek</title><link href="http://example.com/test/cz/translated-article.html" rel="alternate"></link><updated>2014-09-15T00:00:00+00:00</updated><author><name>Test Testovič</name></author><id>tag:example.com,2014-09-15:test/cz/translated-article.html</id><summary type="html">&lt;p&gt;Jednoduchý článek s překlady.
+Zde je odkaz na &lt;a class="reference external" href="http://example.com/test/cz/../images/img.png"&gt;nějaký obrázek&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>Ein übersetzter Artikel</title><link href="http://example.com/test/cz/../de/translated-article.html" rel="alternate"></link><updated>2014-09-14T00:00:00+00:00</updated><author><name>Test Testovič</name></author><id>tag:example.com,2014-09-14:test/cz/../de/translated-article.html</id><summary type="html">&lt;p&gt;Ein einfacher Artikel mit einer Übersetzung.
+Hier ist ein Link zur &lt;a class="reference external" href="http://example.com/test/cz/../images/img.png"&gt;einigem Bild&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>A translated article</title><link href="http://example.com/test/cz/../translated-article.html" rel="alternate"></link><updated>2014-09-13T00:00:00+00:00</updated><author><name>Test Testovič</name></author><id>tag:example.com,2014-09-13:test/cz/../translated-article.html</id><summary type="html">&lt;p&gt;A simple article with a translation.
+Here is a link to &lt;a class="reference external" href="http://example.com/test/cz/../images/img.png"&gt;some image&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>An untranslated article</title><link href="http://example.com/test/cz/an-untranslated-article-en.html" rel="alternate"></link><updated>2014-07-14T00:00:00+00:00</updated><author><name>Test Testovič</name></author><id>tag:example.com,2014-07-14:test/cz/an-untranslated-article-en.html</id><summary type="html">&lt;p&gt;An article without a translation.
+Here is a link to an &lt;a class="reference external" href="http://example.com/test/cz/../pages/untranslated-page.html"&gt;untranslated page&lt;/a&gt;&lt;/p&gt;
+</summary></entry></feed>

test_data/output/cz/index.html

@@ -0,0 +1,54 @@
+<!DOCTYPE html>
+<html lang="cz">
+<head>
+        <title>Welcome to our Testovací stránka</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testovací stránka Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/cz/../theme/style.css" />
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/cz/">Testovací stránka <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+        </ul></nav><!-- /#menu -->
+<section id="content">
+<h2>All articles</h2>
+
+<ol id="post-list">
+        <li><article class="hentry">
+                <header> <h2 class="entry-title"><a href="http://example.com/test/cz/translated-article.html" rel="bookmark" title="Permalink to Přeložený článek">Přeložený článek</a></h2> </header>
+                <footer class="post-info">
+                    <abbr class="published" title="2014-09-15T00:00:00"> Mon 15 September 2014 </abbr>
+                    <address class="vcard author">By
+                        <a class="url fn" href="http://example.com/test/cz/author/test-testovic.html">Test Testovič</a>
+                    </address>
+                </footer><!-- /.post-info -->
+                <div class="entry-content"> <p>Jednoduchý článek s překlady.
+Zde je odkaz na <a class="reference external" href="http://example.com/test/cz/../images/img.png">nějaký obrázek</a>.</p>
+ </div><!-- /.entry-content -->
+        </article></li>
+        <li><article class="hentry">
+                <header> <h2 class="entry-title"><a href="http://example.com/test/cz/an-untranslated-article-en.html" rel="bookmark" title="Permalink to An untranslated article">An untranslated article</a></h2> </header>
+                <footer class="post-info">
+                    <abbr class="published" title="2014-07-14T00:00:00"> Mon 14 July 2014 </abbr>
+                    <address class="vcard author">By
+                        <a class="url fn" href="http://example.com/test/cz/author/test-testovic.html">Test Testovič</a>
+                    </address>
+                </footer><!-- /.post-info -->
+                <div class="entry-content"> <p>An article without a translation.
+Here is a link to an <a class="reference external" href="http://example.com/test/cz/../pages/untranslated-page.html">untranslated page</a></p>
+ </div><!-- /.entry-content -->
+        </article></li>
+</ol><!-- /#posts-list -->
+</section><!-- /#content -->
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/cz/translated-article.html

@@ -0,0 +1,52 @@
+<!DOCTYPE html>
+<html lang="cz">
+<head>
+          <title>Welcome to our Testovací stránka</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testovací stránka Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/cz/../theme/style.css" />
+
+
+
+
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/cz/">Testovací stránka <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+        </ul></nav><!-- /#menu -->
+<section id="content" class="body">
+  <header>
+    <h2 class="entry-title">
+      <a href="http://example.com/test/cz/translated-article.html" rel="bookmark"
+         title="Permalink to Přeložený článek">Přeložený článek</a></h2>
+ Translations: 
+<a href="http://example.com/test/cz/../de/translated-article.html">de</a>
+<a href="http://example.com/test/cz/../translated-article.html">en</a>
+
+  </header>
+  <footer class="post-info">
+    <abbr class="published" title="2014-09-15T00:00:00">
+      Mon 15 September 2014
+    </abbr>
+    <address class="vcard author">
+      By           <a class="url fn" href="http://example.com/test/cz/author/test-testovic.html">Test Testovič</a>
+    </address>
+  </footer><!-- /.post-info -->
+  <div class="entry-content">
+    <p>Jednoduchý článek s překlady.
+Zde je odkaz na <a class="reference external" href="http://example.com/test/cz/../images/img.png">nějaký obrázek</a>.</p>
+
+  </div><!-- /.entry-content -->
+</section>
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/de/drafts/an-untranslated-article-en.html

@@ -0,0 +1,49 @@
+<!DOCTYPE html>
+<html lang="de">
+<head>
+          <title>Willkommen Sie zur unserer Testseite</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testseite Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/de/../theme/style.css" />
+
+
+
+
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/de/">Testseite <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+        </ul></nav><!-- /#menu -->
+<section id="content" class="body">
+  <header>
+    <h2 class="entry-title">
+      <a href="http://example.com/test/de/drafts/an-untranslated-article-en.html" rel="bookmark"
+         title="Permalink to An untranslated article">An untranslated article</a></h2>
+ 
+  </header>
+  <footer class="post-info">
+    <abbr class="published" title="2014-07-14T00:00:00">
+      Mo 14 Juli 2014
+    </abbr>
+    <address class="vcard author">
+      By           <a class="url fn" href="http://example.com/test/de/author/der-tester.html">Der Tester</a>
+    </address>
+  </footer><!-- /.post-info -->
+  <div class="entry-content">
+    <p>An article without a translation.
+Here is a link to an <a class="reference external" href="http://example.com/test/de/pages/untranslated-page-en.html">untranslated page</a></p>
+
+  </div><!-- /.entry-content -->
+</section>
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/de/feeds_all.atom.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Testseite</title><link href="http://example.com/test/de/" rel="alternate"></link><link href="http://example.com/test/feeds_all.atom.xml" rel="self"></link><id>http://example.com/test/de/</id><updated>2014-09-15T00:00:00+00:00</updated><entry><title>Přeložený článek</title><link href="http://example.com/test/de/../cz/translated-article.html" rel="alternate"></link><updated>2014-09-15T00:00:00+00:00</updated><author><name>Der Tester</name></author><id>tag:example.com,2014-09-15:test/de/../cz/translated-article.html</id><summary type="html">&lt;p&gt;Jednoduchý článek s překlady.
+Zde je odkaz na &lt;a class="reference external" href="http://example.com/test/de/../images/img.png"&gt;nějaký obrázek&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>Ein übersetzter Artikel</title><link href="http://example.com/test/de/translated-article.html" rel="alternate"></link><updated>2014-09-14T00:00:00+00:00</updated><author><name>Der Tester</name></author><id>tag:example.com,2014-09-14:test/de/translated-article.html</id><summary type="html">&lt;p&gt;Ein einfacher Artikel mit einer Übersetzung.
+Hier ist ein Link zur &lt;a class="reference external" href="http://example.com/test/de/../images/img.png"&gt;einigem Bild&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>A translated article</title><link href="http://example.com/test/de/../translated-article.html" rel="alternate"></link><updated>2014-09-13T00:00:00+00:00</updated><author><name>Der Tester</name></author><id>tag:example.com,2014-09-13:test/de/../translated-article.html</id><summary type="html">&lt;p&gt;A simple article with a translation.
+Here is a link to &lt;a class="reference external" href="http://example.com/test/de/../images/img.png"&gt;some image&lt;/a&gt;.&lt;/p&gt;
+</summary></entry></feed>

test_data/output/de/index.html

@@ -0,0 +1,42 @@
+<!DOCTYPE html>
+<html lang="de">
+<head>
+        <title>Willkommen Sie zur unserer Testseite</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testseite Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/de/../theme/style.css" />
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/de/">Testseite <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+        </ul></nav><!-- /#menu -->
+<section id="content">
+<h2>All articles</h2>
+
+<ol id="post-list">
+        <li><article class="hentry">
+                <header> <h2 class="entry-title"><a href="http://example.com/test/de/translated-article.html" rel="bookmark" title="Permalink to Ein übersetzter Artikel">Ein übersetzter Artikel</a></h2> </header>
+                <footer class="post-info">
+                    <abbr class="published" title="2014-09-14T00:00:00"> So 14 September 2014 </abbr>
+                    <address class="vcard author">By
+                        <a class="url fn" href="http://example.com/test/de/author/der-tester.html">Der Tester</a>
+                    </address>
+                </footer><!-- /.post-info -->
+                <div class="entry-content"> <p>Ein einfacher Artikel mit einer Übersetzung.
+Hier ist ein Link zur <a class="reference external" href="http://example.com/test/de/../images/img.png">einigem Bild</a>.</p>
+ </div><!-- /.entry-content -->
+        </article></li>
+</ol><!-- /#posts-list -->
+</section><!-- /#content -->
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/de/pages/untranslated-page-en.html

@@ -0,0 +1,30 @@
+<!DOCTYPE html>
+<html lang="de">
+<head>
+        <title>Untranslated page</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testseite Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/de/../theme/style.css" />
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/de/">Testseite <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+        </ul></nav><!-- /#menu -->
+    <h1>Untranslated page</h1>
+    
+
+    <p>This page has no translation.</p>
+
+
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/de/translated-article.html

@@ -0,0 +1,52 @@
+<!DOCTYPE html>
+<html lang="de">
+<head>
+          <title>Willkommen Sie zur unserer Testseite</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testseite Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/de/../theme/style.css" />
+
+
+
+
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/de/">Testseite <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+        </ul></nav><!-- /#menu -->
+<section id="content" class="body">
+  <header>
+    <h2 class="entry-title">
+      <a href="http://example.com/test/de/translated-article.html" rel="bookmark"
+         title="Permalink to Ein übersetzter Artikel">Ein übersetzter Artikel</a></h2>
+ Translations: 
+<a href="http://example.com/test/de/../cz/translated-article.html">cz</a>
+<a href="http://example.com/test/de/../translated-article.html">en</a>
+
+  </header>
+  <footer class="post-info">
+    <abbr class="published" title="2014-09-14T00:00:00">
+      So 14 September 2014
+    </abbr>
+    <address class="vcard author">
+      By           <a class="url fn" href="http://example.com/test/de/author/der-tester.html">Der Tester</a>
+    </address>
+  </footer><!-- /.post-info -->
+  <div class="entry-content">
+    <p>Ein einfacher Artikel mit einer Übersetzung.
+Hier ist ein Link zur <a class="reference external" href="http://example.com/test/de/../images/img.png">einigem Bild</a>.</p>
+
+  </div><!-- /.entry-content -->
+</section>
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/feeds_all.atom.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Testing site</title><link href="http://example.com/test/" rel="alternate"></link><link href="http://example.com/test/feeds_all.atom.xml" rel="self"></link><id>http://example.com/test/</id><updated>2014-09-15T00:00:00+00:00</updated><entry><title>Přeložený článek</title><link href="http://example.com/test/cz/translated-article.html" rel="alternate"></link><updated>2014-09-15T00:00:00+00:00</updated><author><name>The Tester</name></author><id>tag:example.com,2014-09-15:test/cz/translated-article.html</id><summary type="html">&lt;p&gt;Jednoduchý článek s překlady.
+Zde je odkaz na &lt;a class="reference external" href="http://example.com/test/images/img.png"&gt;nějaký obrázek&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>Ein übersetzter Artikel</title><link href="http://example.com/test/de/translated-article.html" rel="alternate"></link><updated>2014-09-14T00:00:00+00:00</updated><author><name>The Tester</name></author><id>tag:example.com,2014-09-14:test/de/translated-article.html</id><summary type="html">&lt;p&gt;Ein einfacher Artikel mit einer Übersetzung.
+Hier ist ein Link zur &lt;a class="reference external" href="http://example.com/test/images/img.png"&gt;einigem Bild&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>A translated article</title><link href="http://example.com/test/translated-article.html" rel="alternate"></link><updated>2014-09-13T00:00:00+00:00</updated><author><name>The Tester</name></author><id>tag:example.com,2014-09-13:test/translated-article.html</id><summary type="html">&lt;p&gt;A simple article with a translation.
+Here is a link to &lt;a class="reference external" href="http://example.com/test/images/img.png"&gt;some image&lt;/a&gt;.&lt;/p&gt;
+</summary></entry><entry><title>An untranslated article</title><link href="http://example.com/test/an-untranslated-article.html" rel="alternate"></link><updated>2014-07-14T00:00:00+00:00</updated><author><name>The Tester</name></author><id>tag:example.com,2014-07-14:test/an-untranslated-article.html</id><summary type="html">&lt;p&gt;An article without a translation.
+Here is a link to an &lt;a class="reference external" href="http://example.com/test/pages/untranslated-page.html"&gt;untranslated page&lt;/a&gt;&lt;/p&gt;
+</summary></entry></feed>

test_data/output/index.html

@@ -0,0 +1,55 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+        <title>Welcome to our Testing site</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testing site Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/theme/style.css" />
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/">Testing site <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+            <li><a href="http://example.com/test/pages/untranslated-page.html">Untranslated page</a></li>
+        </ul></nav><!-- /#menu -->
+<section id="content">
+<h2>All articles</h2>
+
+<ol id="post-list">
+        <li><article class="hentry">
+                <header> <h2 class="entry-title"><a href="http://example.com/test/translated-article.html" rel="bookmark" title="Permalink to A translated article">A translated article</a></h2> </header>
+                <footer class="post-info">
+                    <abbr class="published" title="2014-09-13T00:00:00"> Sat 13 September 2014 </abbr>
+                    <address class="vcard author">By
+                        <a class="url fn" href="http://example.com/test/author/the-tester.html">The Tester</a>
+                    </address>
+                </footer><!-- /.post-info -->
+                <div class="entry-content"> <p>A simple article with a translation.
+Here is a link to <a class="reference external" href="http://example.com/test/images/img.png">some image</a>.</p>
+ </div><!-- /.entry-content -->
+        </article></li>
+        <li><article class="hentry">
+                <header> <h2 class="entry-title"><a href="http://example.com/test/an-untranslated-article.html" rel="bookmark" title="Permalink to An untranslated article">An untranslated article</a></h2> </header>
+                <footer class="post-info">
+                    <abbr class="published" title="2014-07-14T00:00:00"> Mon 14 July 2014 </abbr>
+                    <address class="vcard author">By
+                        <a class="url fn" href="http://example.com/test/author/the-tester.html">The Tester</a>
+                    </address>
+                </footer><!-- /.post-info -->
+                <div class="entry-content"> <p>An article without a translation.
+Here is a link to an <a class="reference external" href="http://example.com/test/pages/untranslated-page.html">untranslated page</a></p>
+ </div><!-- /.entry-content -->
+        </article></li>
+</ol><!-- /#posts-list -->
+</section><!-- /#content -->
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/pages/untranslated-page.html

@@ -0,0 +1,31 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+        <title>Untranslated page</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testing site Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/theme/style.css" />
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/">Testing site <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+            <li class="active"><a href="http://example.com/test/pages/untranslated-page.html">Untranslated page</a></li>
+        </ul></nav><!-- /#menu -->
+    <h1>Untranslated page</h1>
+    
+
+    <p>This page has no translation.</p>
+
+
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/output/translated-article.html

@@ -0,0 +1,53 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+          <title>Welcome to our Testing site</title>
+        <meta charset="utf-8" />
+        <link href="http://example.com/test/feeds_all.atom.xml" type="application/atom+xml" rel="alternate" title="Testing site Full Atom Feed" />
+
+<link rel="stylesheet" href="http://example.com/test/theme/style.css" />
+
+
+
+
+</head>
+
+<body id="index" class="home">
+        <header id="banner" class="body">
+                <h1><a href="http://example.com/test/">Testing site <strong></strong></a></h1>
+        </header><!-- /#banner -->
+        <nav id="menu"><ul>
+            <li><a href="http://example.com/test/pages/untranslated-page.html">Untranslated page</a></li>
+        </ul></nav><!-- /#menu -->
+<section id="content" class="body">
+  <header>
+    <h2 class="entry-title">
+      <a href="http://example.com/test/translated-article.html" rel="bookmark"
+         title="Permalink to A translated article">A translated article</a></h2>
+ Translations: 
+<a href="http://example.com/test/cz/translated-article.html">cz</a>
+<a href="http://example.com/test/de/translated-article.html">de</a>
+
+  </header>
+  <footer class="post-info">
+    <abbr class="published" title="2014-09-13T00:00:00">
+      Sat 13 September 2014
+    </abbr>
+    <address class="vcard author">
+      By           <a class="url fn" href="http://example.com/test/author/the-tester.html">The Tester</a>
+    </address>
+  </footer><!-- /.post-info -->
+  <div class="entry-content">
+    <p>A simple article with a translation.
+Here is a link to <a class="reference external" href="http://example.com/test/images/img.png">some image</a>.</p>
+
+  </div><!-- /.entry-content -->
+</section>
+        <footer id="contentinfo" class="body">
+                <address id="about" class="vcard body">
+                Proudly powered by <a href="http://getpelican.com/">Pelican</a>,
+                which takes great advantage of <a href="http://python.org">Python</a>.
+                </address><!-- /#about -->
+        </footer><!-- /#contentinfo -->
+</body>
+</html>

test_data/pelicanconf.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*- #
+from __future__ import unicode_literals
+
+AUTHOR = 'The Tester'
+SITENAME = 'Testing site'
+SITEURL = 'http://example.com/test'
+
+# to make the test suite portable
+TIMEZONE = 'UTC'
+
+DEFAULT_LANG = 'en'
+LOCALE = 'en_US.UTF-8'
+
+# Generate only one feed
+FEED_ALL_ATOM = 'feeds_all.atom.xml'
+CATEGORY_FEED_ATOM = None
+TRANSLATION_FEED_ATOM = None
+AUTHOR_FEED_ATOM = None
+AUTHOR_FEED_RSS = None
+
+# Disable unnecessary pages
+CATEGORY_SAVE_AS = ''
+TAG_SAVE_AS = ''
+AUTHOR_SAVE_AS = ''
+ARCHIVES_SAVE_AS = ''
+AUTHORS_SAVE_AS = ''
+CATEGORIES_SAVE_AS = ''
+TAGS_SAVE_AS = ''
+
+PLUGIN_PATHS = ['../../']
+PLUGINS = ['i18n_subsites']
+
+THEME = 'localized_theme'
+JINJA_EXTENSIONS = ['jinja2.ext.i18n']
+
+from blinker import signal
+tmpsig = signal('tmpsig')
+I18N_FILTER_SIGNALS = [tmpsig]
+
+I18N_SUBSITES = {
+    'de': {
+        'SITENAME': 'Testseite',
+        'AUTHOR': 'Der Tester',
+        'LOCALE': 'de_DE.UTF-8',
+        },
+    'cz': {
+        'SITENAME': 'Testovací stránka',
+        'AUTHOR': 'Test Testovič',
+        'I18N_UNTRANSLATED_PAGES': 'remove',
+        'I18N_UNTRANSLATED_ARTICLES': 'keep',
+        },
+    }

test_i18n_subsites.py

@@ -0,0 +1,139 @@
+'''Unit tests for the i18n_subsites plugin'''
+
+import os
+import locale
+import unittest
+import subprocess
+from tempfile import mkdtemp
+from shutil import rmtree
+
+from . import i18n_subsites as i18ns
+from pelican import Pelican
+from pelican.tests.support import get_settings
+from pelican.settings import read_settings
+
+
+class TestTemporaryLocale(unittest.TestCase):
+    '''Test the temporary locale context manager'''
+
+    def test_locale_restored(self):
+        '''Test that the locale is restored after exiting context'''
+        orig_locale = locale.setlocale(locale.LC_ALL)
+        with i18ns.temporary_locale():
+            locale.setlocale(locale.LC_ALL, 'C')
+            self.assertEqual(locale.setlocale(locale.LC_ALL), 'C')
+        self.assertEqual(locale.setlocale(locale.LC_ALL), orig_locale)
+
+    def test_temp_locale_set(self):
+        '''Test that the temporary locale is set'''
+        with i18ns.temporary_locale('C'):
+            self.assertEqual(locale.setlocale(locale.LC_ALL), 'C')
+
+
+class TestSettingsManipulation(unittest.TestCase):
+    '''Test operations on settings dict'''
+
+    def setUp(self):
+        '''Prepare default settings'''
+        self.settings = get_settings()
+
+    def test_get_pelican_cls_class(self):
+        '''Test that we get class given as an object'''
+        self.settings['PELICAN_CLASS'] = object
+        cls = i18ns.get_pelican_cls(self.settings)
+        self.assertIs(cls, object)
+        
+    def test_get_pelican_cls_str(self):
+        '''Test that we get correct class given by string'''
+        cls = i18ns.get_pelican_cls(self.settings)
+        self.assertIs(cls, Pelican)
+        
+
+class TestSitesRelpath(unittest.TestCase):
+    '''Test relative path between sites generation'''
+
+    def setUp(self):
+        '''Generate some sample siteurls'''
+        self.siteurl = 'http://example.com'
+        i18ns._SITE_DB['en'] = self.siteurl
+        i18ns._SITE_DB['de'] = self.siteurl + '/de'
+
+    def tearDown(self):
+        '''Remove sites from db'''
+        i18ns._SITE_DB.clear()
+
+    def test_get_site_path(self):
+        '''Test getting the path within a site'''
+        self.assertEqual(i18ns.get_site_path(self.siteurl), '/')
+        self.assertEqual(i18ns.get_site_path(self.siteurl + '/de'), '/de')
+
+    def test_relpath_to_site(self):
+        '''Test getting relative paths between sites'''
+        self.assertEqual(i18ns.relpath_to_site('en', 'de'), 'de')
+        self.assertEqual(i18ns.relpath_to_site('de', 'en'), '..')
+
+        
+class TestRegistration(unittest.TestCase):
+    '''Test plugin registration'''
+
+    def test_return_on_missing_signal(self):
+        '''Test return on missing required signal'''
+        i18ns._SIGNAL_HANDLERS_DB['tmp_sig'] = None
+        i18ns.register()
+        self.assertNotIn(id(i18ns.save_generator),
+                         i18ns.signals.generator_init.receivers)
+
+    def test_registration(self):
+        '''Test registration of all signal handlers'''
+        i18ns.register()
+        for sig_name, handler in i18ns._SIGNAL_HANDLERS_DB.items():
+            sig = getattr(i18ns.signals, sig_name)
+            self.assertIn(id(handler), sig.receivers)
+            # clean up
+            sig.disconnect(handler)
+        
+
+class TestFullRun(unittest.TestCase):
+    '''Test running Pelican with the Plugin'''
+
+    def setUp(self):
+        '''Create temporary output and cache folders'''
+        self.temp_path = mkdtemp(prefix='pelicantests.')
+        self.temp_cache = mkdtemp(prefix='pelican_cache.')
+
+    def tearDown(self):
+        '''Remove output and cache folders'''
+        rmtree(self.temp_path)
+        rmtree(self.temp_cache)
+
+    def test_sites_generation(self):
+        '''Test generation of sites with the plugin
+
+        Compare with recorded output via ``git diff``.
+        To generate output for comparison run the command
+        ``pelican -o test_data/output -s test_data/pelicanconf.py \
+        test_data/content``
+        Remember to remove the output/ folder before that.
+        '''
+        base_path = os.path.dirname(os.path.abspath(__file__))
+        base_path = os.path.join(base_path, 'test_data')
+        content_path = os.path.join(base_path, 'content')
+        output_path = os.path.join(base_path, 'output')
+        settings_path = os.path.join(base_path, 'pelicanconf.py')
+        settings = read_settings(path=settings_path, override={
+            'PATH': content_path,
+            'OUTPUT_PATH': self.temp_path,
+            'CACHE_PATH': self.temp_cache,
+            'PLUGINS': [i18ns],
+            }
+        )
+        pelican = Pelican(settings)
+        pelican.run()
+
+        # compare output
+        out, err = subprocess.Popen(
+            ['git', 'diff', '--no-ext-diff', '--exit-code', '-w', output_path,
+             self.temp_path], env={'PAGER': ''},
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+        self.assertFalse(out, 'non-empty `diff` stdout:\n{}'.format(out))
+        self.assertFalse(err, 'non-empty `diff` stderr:\n{}'.format(out))