<sect1 id="synchronisation-scripts">
<title>Translation synchronisation</title>

<sect2>
<title>
The need for a synchronization process
</title>

<para>
The files which are really used for building &d-i; packages are hosted
in the <acronym>git</acronym> repository used for the development of &d-i;
packages. Each &d-i; package sources is hosted in a subdirectory of
the <filename>packages/</filename> directory, with a very classical
Debian package source tree organization, including the
<filename>debian/po</filename> directory where the package's
translations are hosted.
</para>

<para>
These files need to be synchronised with the so-called
<emphasis>master</emphasis> files from the packages/po directory, since
translators only work on these master files. This
synchronization must work both ways with changes on the English original strings
going from individual packages to the master files and translation changes/updates
from the master files going to individual packages:
</para>

<itemizedlist>
  <listitem>
    <para>
      When &d-i; packages templates are changed, the individual
      <filename>debian/po/templates.pot</filename> file will
      change. These changes have to be moved to files in
      <filename>packages/po</filename>;
    </para>
  </listitem>
  <listitem>
    <para>
      When translations are updated in the
      <filename>packages/po</filename> directory, these changes have
      to be moved back into individual
      <filename>debian/po/*.po</filename> files for each package. This
      does not need to be done at each translation update, but must be
      done <emphasis>before</emphasis> building/uploading packages.
  </para>
  </listitem>
</itemizedlist>

<para>
The task of keeping track of the correct synchronisation is one of the
most important tasks of the &i18n-coords;. It is handled by the
<command>scripts/l10n/l10n-sync</command> script.
</para>

</sect2>

<sect2>
<title>
Localization files synchronisation process
</title>

<sect3>
<title>
Synchronization process
</title>

<para>
The <command>l10n-sync</command> script is run periodically under the "D-I role"
account by a <ulink
url="https://salsa.debian.org/installer-team/d-i/-/blob/master/scripts/etc/cron/crontab">cron job</ulink>
on dillon.debian.org:

<informalexample>
<screen>
0 20 * * * cd $DI; mr -q up ; nice -n 19 $DI/scripts/l10n/l10n-sync --levels=5 --online --commit $DI
</screen>
</informalexample>

Its runs are monitored by the &i18n-coords;.
</para>

<para>
It will update the general
<filename>template.pot</filename> and all PO files from individual
<filename>templates.pot</filename> files from all &d-i; packages.
</para>

<para>
While running, this script will also run
<command>debconf-updatepo</command> for each &d-i; package and will
commit back the regenerated templates.pot files (package maintainers
often forget running <command>debconf-udpatepo</command> when
commiting changes to English templates).
</para>

<para>
During normal development, this synchronization process is run once a
day. During release preparations, the frequency might be increased to
speed up the full cycle.
</para>

<para>  
The general process is:
</para>

<orderedlist>
  <listitem>
    <para>
      <emphasis>Initial step</emphasis>: synchronize the whole repository;
    </para>
  </listitem>
  <listitem>
    <para>
      <emphasis>Individual packages update step</emphasis>. For each
      &d-i; package:
    </para>
    <orderedlist>
      <listitem>
        <para>
          synchronize the local copy with the &d-i; <acronym>GIT</acronym>;
        </para>
      </listitem>
      <listitem>
        <para>
          run <command>debconf-updatepo</command>;
        </para>
      </listitem>
      <listitem>
        <para>
          commit back the files in <filename>debian/po</filename>.
        </para>
      </listitem>
    </orderedlist>
  </listitem>
  <listitem>
    <para>
      <emphasis>Master templates file update step</emphasis>:
    </para>
    <orderedlist>
      <listitem>
        <para>
          merge all <filename>templates.pot</filename> files to
          <filename>packages/po/template.pot</filename>.

<!--      The
          <filename>packages/po/header.pot</filename> is mandatory
          (this file will provide the standard header for the merged
          POT file; -->

        </para>
      </listitem>
    </orderedlist>
  </listitem>
  <listitem>
    <para>
      <emphasis>Master PO files update step</emphasis>. For each PO
      file in <filename>packages/po</filename>:
    </para>
    <orderedlist>
      <listitem>
        <para>
          synchronize with &d-i; <acronym>GIT</acronym>;
        </para>
      </listitem>
      <listitem>
        <para>
          if the script is run with the <userinput>--merge</userinput> option,
          merge translations from the reference PO file (the reference file is
          given priority for identical strings);
        </para>
      </listitem>
      <listitem>
        <para>
          update with
          <filename>packages/po/template.pot</filename>. This step
          uses the <command>msgmerge</command> command. The resulting PO file
          formatting details will depend on the version of this
          utility. For that reasons, the synchronization script should
          always be run from machines using the same Debian release
          version (these days it permanently runs on dillon anyway);
        </para>
      </listitem>
      <listitem>
        <para>
          commit back the changed files to &d-i; <acronym>GIT</acronym>.
        </para>
      </listitem>
    </orderedlist>
  </listitem>
  <listitem>
    <para>
      <emphasis>Individual packages PO files update
      step</emphasis>. For each &d-i; package:
    </para>
    <orderedlist>
      <listitem>
        <para>
          synchronize the local copy with the &d-i; <acronym>GIT</acronym> (in case some
          update occurred in the meantime);
        </para>
      </listitem>
      <listitem>
        <para>
          update <filename>debian/po/*.po</filename> files with master
          files;
        </para>
      </listitem>
      <listitem>
        <para>
          commit back the changes to &d-i; <acronym>GIT</acronym>;
        </para>
      </listitem>
    </orderedlist>
  </listitem>
</orderedlist>

<para>
This system minimizes race conditions which could trigger conflicts.
</para>

<para>
The conflict windows have been minimised as much as possible in the
<command>l10n-sync</command> script. However, in order to limit the
number of commits made by the script, files are not commited as soon
as they are modified. So, the
script is sometimes likely to trigger conflicts. For that reason, it
will stop working on a <acronym>GIT</acronym> copy where <acronym>GIT</acronym> conflict files are
present. This explains why this script must always be monitored even
when it is scheduled to be executed periodically.
</para>

</sect3>

<sect3 id="l10n_synchronisation_script">
<title>
Synchronization script use
</title>

<para>
The l10n synchronisation script is kept in &d-i; repository in the
<filename>scripts/l10n/</filename> directory and is named
<filename>l10n-sync</filename>.
</para>

<para>
It is a shell script (which may contain some bashisms until a
skilled shell programmer cleans it out).
</para>

<para>
The script uses some command-line switches which may affect its behaviour.
Some of these switches are mostly present for historical reasons and are kept
because they may have some new use in the future:
</para>


<itemizedlist>
  <listitem>
    <para>
        <userinput>--debug</userinput> will trigger more output by the
        script. Otherwise, the script reports about its actions but
        commands output (such as <command>git</command>,
        <command>debconf-updatepo</command>,
        <command>msgmerge</command>) will be redirected to
        <filename>/dev/null</filename>;
    </para>
  </listitem>
  <listitem>
    <para>
        <userinput>--online</userinput> triggers <command>git
        pull</command> commands before working on files. Except for
        testing, there is no reason for <emphasis>not</emphasis> using
        this switch.
    </para>
  </listitem>
  <listitem>
    <para>
        <userinput>--commit</userinput> allows the script to
        <emphasis>commit</emphasis> files back to the &d-i;
        repository. Otherwise, the modified files are kept in the
        local <acronym>GIT</acronym> copy. Of course, this switch should be used in
        production. Omitting it is useful only for testing.
    </para>
  </listitem>
<!--
  <listitem>
    <para>
        the <userinput>-atomic</userinput> switch instructs
        <command>l10n-sync</command> to commit files as soon as they
        are modified. This makes the script quite slow and may trigger
        dozens of commits. As &d-i; commits are sent to the
        development IRC channel and are followed by several &d-i;
        maintainers, this switch should not be used except in very
        rare occasions;
    </para>
  </listitem>
  <listitem>
    <para>
        the <userinput>-atomic-updates</userinput> switch instructs
        l10n-sync to issue a <command>git pull</command> command
        before working on each package. This may be very CPU, network-
        and time-consuming and should only be used when there is some
        need to limit possible conflicts;
    </para>
  </listitem>
  <listitem>
    <para>
        the <userinput>-keep-revision</userinput> switch needs an
        argument which must be a language code (one only). It will
        force <command>l10n-sync</command> to restore the
        <userinput>PO-Revision-Date</userinput> field of this language
        on each modified file. This switch is used when switching
        languages to the master file (see <xref
        linkend="switch-language"/>. It should
        <emphasis>not</emphasis> be used in other occasions;
    </para>
  </listitem>
-->
  <listitem>
    <para>
        the <userinput>--debconf-updatepo</userinput> defines the
        command for calling the <command>debconf-updatepo</command>
        utility. It allows using a specially crafted
        <command>debconf-updatepo</command> switch, often the
        <userinput>--skip-merge</userinput> switch which only updates
        <filename>templates.pot</filename> files and not PO files
        (using this generally speeds up the synchronization script);
    </para>
  </listitem>
<!--
  <listitem>
    <para>
        the <userinput>-sort-order</userinput> switch allows giving
        at the command line the order the packages should be dealt
        with. This allows to put a crafted order in the generated
        <filename>templates.pot</filename> file so that the
        translators begin to work on the most important packages;
    </para>
  </listitem>
-->
  <listitem>
    <para>
        the <userinput>--merge</userinput> switch allows merging
        master files from another branch. It is used, for instance, on
        the sarge branch for merging translations coming from trunk.
    </para>
  </listitem>
</itemizedlist>

<para>
The script needs the location of the local copy of the &d-i;
repository as an argument. It makes some simple checks about the
copy.
</para>

<para>
When using this script with commits, the &d-i; copy must be as clean as
possible. It should not be used for development tasks. The script
checks for possible <acronym>GIT</acronym> conflict files and aborts if it finds some.
</para>

<para>
Prospective languages (see <xref linkend="prospective"/>)
are handled in a special way: for all such languages, the translations
are not copied in the individual packages directories.
</para>

</sect3>

<sect3>
<title>
The PROSPECTIVE list
</title>

<para>When languages are on early stages of translation after they have been added
through the new language process (<xref linkend="new-language"/>), they are
temporarily listed in a file named <filename>packages/po/PROSPECTIVE</filename>.
</para>

<para>
Languages listed in that file are excluded from synchronization, and,
therefore, PO files are not created in individual packages. As a consequence,
<emphasis>activating</emphasis> a language means that the language code is
<emphasis>removed</emphasis> from the <filename>PROSPECTUVE</filename> file.
</para>

</sect3>

<sect3>
<title>
List of handled packages
</title>

<para>
A special file, named <filename>packages/po/packages_list</filename>,
lists the &d-i; packages which are handled by the
<command>l10n-sync</command> script and are therefore
included in the master files.
<!--
This
file also sorts these packages by order of priority for
translations. The strings at the beginning of the files are to be
translated first.
-->
</para>

<para>
The &d-i; packages maintainers must request for their packages to be
included in this file, when they estimate that their package is ready
for translation. Only &i18n-coords; can integrate packages there, after
checking that the strings have been reviewed.
</para>

</sect3>


<!--
<sect3>
<title>
Synchronization script and automated commits
</title>

<para>
Being aimed to be an automated process with automated commits, the
script requires a few prerequisites to be run without user
interaction.
</para>

<para>
First of all, when run from a cron job, the script has to be able to
commit files. This means that the account it is run from should use a
<acronym>SSH</acronym> key with an empty passphrase and add this key
to the account it commits to on &git-debian;.
</para>

<para>
First, create a <acronym>SSH</acronym> key with an empty passphrase and
put it in a special file:
</para>

<informalexample>
<screen>
user@host:~> ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_dsa): \
        /home/user/.ssh/nopass
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/nopass.
Your public key has been saved in /home/user/.ssh/nopass.pub.
The key fingerprint is:
75:fa:62:54:2c:34:09:96:ad:f2:57:cf:16:ce:74:69 bubulle@mykerinos
</screen>
</informalexample>

<para>
Then, this key should be added to
<filename>~/.ssh/authorized_keys</filename> on &git-debian; for the
Salsa account (&lt;salsa_account&gt;) under which identity commits
will be done.
</para>

<para>
Finally, on the host from which <command>l10n-sync</command> runs will
happen, the following should be added to
<filename>~/.ssh/config</filename>:
</para>

<informalexample>
<screen>
host &git-debian;
  user &lt;salsa_account&gt;
</screen>
</informalexample>

<para>
Of course, this actually means that this local account will then be
able to commit to &git-debian; with &lt;salsa_account&gt; as
identity, <emphasis>without further control</emphasis>. This means
that the account the script is run with should be very safely
protected.
</para>
</sect3>
-->

<sect3>
<title>
Synchronization script control
</title>

<para>
Some mechanisms have been implemented to add more safety and remote
control possibilities to the synchronisation script.
</para>

<para>
First of all, before doing any work, the script opens a special file
kept in the <acronym>GIT</acronym> repository and named
<filename>packages/po/run-l10n-sync</filename>. The "run=" line in
this file mentions whether synchronisation should happen or not. If
the file contains "run=0", then the script will exit without taking
any action.
</para>

<para>
This mechanism gives all &d-i; developers with commit access to the
<acronym>GIT</acronym> repository a very simple way to disable the
synchronization script actions. For this, developers just need to
change the file and commit the new version.
</para>

<para>
While the script is disabled, runs can still be enforced by using the
<command>-force</command> swith, which will ignore the
<filename>packages/po/run-l10n-sync</filename> file.
</para>

<para>
While it is running, the script also creates a file named
<filename>.l10n-sync.lock</filename> at the root of the local copy of
&d-i; repository. The file is removed only after successful runs of
the script. When this file already exists, the script does not run and
exits with an error message.
</para>

<para>
This mechanism prevents running the synchronization script twice on
the same &d-i; repository checkout copy.
</para>

</sect3>

</sect2>

</sect1>
