Community involvement needed: Pale Moon v27+ and localization

The l10n of Pale Moon. Rawr.
User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Community involvement needed: Pale Moon v27+ and localization

Unread post by Moonchild » 2016-10-01, 19:45

I've gone over this a good number of times in my head, and thought about how to move forward with this, come the new milestone.

Looking at everything that needs to be done code-wise, to keep the browser running smooth and secure, neither myself nor the other current core developers will have reasonable amounts of time to dedicate to a localization effort. A few good ideas have been suggested how to approach this by the community, and I'm certainly amenable to it, but it is simply going to be too big of a task to continue doing "on the side" -- even with just the maintenance of updates of language packs here and there, the ball got dropped more than once because of priorities otherwise.
As developers, we're going to have to focus on Pale Moon's primary language above all, and any localization effort will have to have the backing to respond to any added or changed strings relatively quickly.

So, I'm calling on you, our community members, to get involved. I'd like to appoint someone (or a few people, but preferably one organization wizard to direct and delegate) to lead this localization and translation effort; to take care of setting up and maintaining an environment where translators can contribute in a linguist-friendly way; to take care of quality&testing, translation disagreements, and the general tasks involved in creating, packaging and publishing language packs.

Doing this for the browser will be a long term dedication, and will require willingness to learn, be flexible, professional, but above all be responsible. Although voluntary, it will require dedication and a solid promise that you will do your best to make this work, and communicate clearly with me if your circumstances change and/or you have to stop doing this, need help, or have a suggestion for someone else to take over.
I can make the infrastructure available that you might need (a dedicated server, etc., although administration will be your responsibility) and you don't have to do this all for free, either -- although we don't have the budget to hire linguists at normal market rates so it will always have to be in part a form of charity to us.

I fully believe in open communication in this context, so please feel free to step up, volunteer, discuss tasks and practical suggestions to make this effort possible.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

José Labán
Moonbather
Moonbather
Posts: 51
Joined: 2014-10-13, 22:12
Location: Tenochtitlan

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by José Labán » 2016-10-02, 01:43

How do I volunteer? I can help in spanish for Mexico if it's needed.

KNTRO

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by KNTRO » 2016-10-02, 07:08

If I understand this well, then we're back to square one, aren't we? I mean, do we have a l10n platform already to begin with?

Besides, this doesn't seem to be related with translation… only. This is bigger, this is a management task mainly. It's even more technical than linguistic.

I'm wondering if I could be a little helpful here, somehow. Could you, please, enumerate the different localization tasks for Pale Moon 27+, and how many people are required in each task?

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Moonchild » 2016-10-02, 09:49

KNTRO wrote:If I understand this well, then we're back to square one, aren't we? I mean, do we have a l10n platform already to begin with?
You know we don't, and we haven't had one for a while.
One of the main tasks of whomever wants to lead this effort is to find and make available such a platform, as I've clearly said in my OP here.
KNTRO wrote:Besides, this doesn't seem to be related with translation… only. This is bigger, this is a management task mainly. It's even more technical than linguistic.
I thought that much was clear? If not, then yes, I'll confirm, once more, next to translators we above all need someone to lead this effort; to set up, to organize, to manage and to do the technical/administrative side of things. We simply don't have the time/capacity to focus on this ourselves, it needs a dedicated person to make this happen.
KNTRO wrote:Could you, please, enumerate the different localization tasks for Pale Moon 27+, and how many people are required in each task?
Not many people will be needed to get this going, but I'll try to break it down:
  • Someone to act as lead for this effort. They will be the primary contact with the rest of the developers.
    This person will be in charge of organizing communication with translators and have the options to create/publish language packs.
  • Someone with the technical know-how to find, set up, implement and maintain a localization framework of some sort that is compatible with our formats and that is linguist-friendly enough for translators to work in. This can be the same person as the previous point, depending on skills.
  • For each proposed language: at the lead's discretion. I would suggest attracting at least 2 people per individual language, with a native speaker doing proofreading.
As said, I can make infrastructure and resources available as-needed (within reason) to make the second point possible and rewards are going to be available for people investing time, effort and expertise into this effort.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Sob__
Lunatic
Lunatic
Posts: 251
Joined: 2014-02-17, 01:12
Location: CZ

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Sob__ » 2016-10-03, 23:49

Stupid question, sorry if I missed something obvious. I assume it will be the same kind of localization as before, i.e. dtd and properties files, right? Is there any documentation how to take those files from main source tree and create langpack from them? I'm looking at current PM26 sources for reference how it's done and it doesn't make much sense so far. Those files are all over the place in main source tree, and they get somehow put in completely different directories to form a langpack. Is there any key to that? I mean, how exactly this happens:

\browser\locales\en-US\chrome\browser\browser.dtd -> \browser\chrome\AB-CD\locale\browser\browser.dtd
\browser\locales\en-US\chrome\overrides\netError.dtd -> \browser\chrome\AB-CD\locale\browser\netError.dtd
\dom\locales\en-US\chrome\netError.dtd -> \chrome\AB-CD\locale\AB-CD\global\netError.dtd
\security\manager\locales\en-US\chrome\pippki\pippki.dtd -> \chrome\AB-CD\locale\AB-CD\pippki\pippki.dtd
\some\different\path\locales\en-US\something\file.dtd -> ???

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Moonchild » 2016-10-04, 08:08

Yes, the basic setup won't be any different than what we have now; but the actual entities and strings will be significantly different.
The locale files are indeed spread out in the source tree. This is in part because of our multi-tiered design (strings for, say, the browser are irrelevant for the mail client based on the same, but strings in the toolkit are relevant).

When building, all these files are gathered up and processed, and they end up in a ready-to-use en-US language pack in $obj-dir/dist -- that language pack can serve as a "source" for localization efforts; all strings that are in there should be in other language packs for them to work.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Sob__
Lunatic
Lunatic
Posts: 251
Joined: 2014-02-17, 01:12
Location: CZ

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Sob__ » 2016-10-04, 15:47

Can you please share this en-US langpack for current PM 27 beta 1? I'd try to build my own, but since these nice step by step instructions are only for older version, with explicit warning about new one, I guess it might not yet be as simple as before.

My rough idea is this: I don't believe you about those significant changes. ;) Don't take me wrong, of course you know best what happened. But when I look at dtd files found in omni.ja, many didn't change at all. And even those that did change, still have lot of unchanged entities and strings. So I think it shouldn't be too hard to make few scripts, parse old and new en-US files and old localized files, mix it together and spit out working partial localization, which should be nice starting point for further efforts.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Moonchild » 2016-10-04, 15:58

Sob__ wrote:My rough idea is this: I don't believe you about those significant changes. ;) Don't take me wrong, of course you know best what happened. But when I look at dtd files found in omni.ja, many didn't change at all. And even those that did change, still have lot of unchanged entities and strings. So I think it shouldn't be too hard to make few scripts, parse old and new en-US files and old localized files, mix it together and spit out working partial localization, which should be nice starting point for further efforts.
I never said that everything had changed. Of course there's going to be a lot of overlap since we're still based on Mozilla. But there are also going to be a lot of changes.
But what I've been saying is that these good practical ideas have to be organized, and I simply can't do this, nor other people who focus on the core code. Our plates are full.

Attached a beta1 en-US pack.
Attachments
palemoon-27.0.0b1.en-US.langpack.xpi
(389.89 KiB) Downloaded 63 times
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

New Tobin Paradigm

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by New Tobin Paradigm » 2016-10-04, 16:04

Okay.. Here is what happened.. As you should know, because I have in-depth explained it over and over again.. Tycho, VERY SIMPLY PUT, is the result of forking a newer Mozilla platform.. Which has it's own subset of l10n strings mostly in /toolkit/locales. We deleted any trace of Firefox save devtools and copied over /browser from current source code to the new platform which has it's own l10n in /browser/locales. We have already diverged in the browser's l10n.. Merged in the older palemoon.dtd and added strings for various reasons.. So older langpacks for the browser parts of l10n are already out of date. Since we have a new platform the platform l10n bits are also horribly out of date. THUS you COULD theoretically mangle browser and platform's historical ancester's l10n from Pale Moon 26 and from Tycho's platform THEN place all the missing strings in.. And then do it 30 more times.. The new strings, however, aren't translated.

NOW, when the browser is built.. it spits out an en-US langpack which is a composite of several locations in the codebase but MOSTLY from browser and toolkit's locales directories..

At any rate trying to mangle l10n from previous langpacks and Tycho's historical ancestor as well as adding in the missing strings will require at least one manual audit and then an automation solution would need to be devised and THEN would have to be repeated for all 30. THEN we would need an additional audit of the end result.

OR we can start from en-US and just do it properly.. You can chose which of the two is harder.

Additionally, I believe whomever would be in charge of this should focus on 5 or so core languages. Also, I believe producing langpacks at this stage of development should not be attempted. However, prep work getting all your ducks in a row can be. I assure you there could be more l10n changes before final release.

Also, allow me to remind you that ANY mistake in a langpack will bust the application in some spectacular ways.. So everything MUST be JUST right!

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Moonchild » 2016-10-04, 17:26

Matt A Tobin wrote:I believe whomever would be in charge of this should focus on 5 or so core languages.
Those being the most commonly-known languages. French, one generic flavor of Spanish, one generic flavor of Portuguese, and German.
Considering we have a fairly large following in Japan and Russian-speaking countries, that would be two more candidates.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

GMforker

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by GMforker » 2016-10-04, 18:17

Moonchild wrote:
Matt A Tobin wrote:I believe whomever would be in charge of this should focus on 5 or so core languages.
Those being the most commonly-known languages. French, one generic flavor of Spanish, one generic flavor of Portuguese, and German.
Considering we have a fairly large following in Japan and Russian-speaking countries, that would be two more candidates.
Czech users will be happy :twisted:
(I'm just trying to bring a little humor)

riiis
Lunatic
Lunatic
Posts: 471
Joined: 2014-05-17, 15:51
Location: USA

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by riiis » 2016-10-04, 22:11

Producing all 33 Language Packs for Tycho should not be difficult or time-consuming. Because Tycho is viewed as difficult and complex-- this does not mean that Language Packs are, in reality, difficult or complex (in fact, to the contrary, Language Packs indeed are neither difficult nor complex). After all, Language Packs consist primarily of simple text files, nothing more, which text files happen to end in ".dtd" or ".properties".

The "properties" files in Tycho Alpha2 contain 4538 properties strings, of which only 752 new strings (16.6%) were added in Tycho, and 3786 properties strings (83.4%) were carried forward from Pale Moon 26. The "dtd" files in Tycho Alpha2 contain 3355 entities strings, of which only 177 new strings (5.3%) were added in Tycho, and 3178 entities strings (97.7%) were carried forward from Pale Moon 26. These statistics hardly justify zero-base budgeting, zero-base translations, zero-base Language Packs, or zero-base anything else.

All 33 new Language Packs for Tycho should be produced all at one time, using a concurrent process. If you believe that Language Packs are too difficult or esoteric to produce concurrently (but instead must be produced one or a few Language Packs at a time), you're probably doing things wrong.

All that is required to produce all 33 Language Packs for Tycho is a team of individuals with computers and minimum spreadsheet skills. There are two "omni.ja" files in Tycho, each consisting of two types of language files ("dtd" files and "properties" files). So, there are basicly 4 groups of data that must be processed separately.

Step 1: List the entities and properties strings from the two omni.ja files on 4 separate worksheets. List the entities and properties strings from corresponding Language Packs files on separate worksheets. Use the spreadsheet function "VLOOKUP" (set to find an exact match) to look up, in the Tycho lists) all entities and properties strings currently in the Language Packs lists. For all strings in the Language Packs lists, not in the Tycho lists, VLOOKUP returns an error message (#N/A). Remove these records from the Language Packs (these strings are no longer part of Tycho). What's left in the Language Packs are just those strings carried forward from Pale Moon 26 to Pale Moon 27 (Tycho). Copy these strings from the spreadsheet and paste to the respective Language Pack files (all items in the Language Pack should be replaced).

Step 2: Use "VLOOKUP" to look up, this time in the Language Packs lists, all entities and properties strings currently in the Tycho lists. For all strings in the Tycho lists, not in the Language Packs lists, these are new strings also needing to be copied and pasted to the respective Language Packs files.

Step 3: During VLOOKUP, the Language Pack worksheet examples, below, retrieve the English string values from the Tycho lists, which values can be matched to the non-English string values already in the Language Pack. If the "non-English" string matches the English string from the Tycho lists, the string may require translation. If so, these items could be merged with the items in step 2 above. Then, the merged list could be sent to GitHub for translation.

4. Fourth, the Language Packs will now work in Tycho, but with some, perhaps many English strings substituted for non-English strings. The prevalence of these English strings can be evaluated, prior to either 1) releasing the Language Pack or 2) holding the Language Pack for additional translations.
dtd-worksheet.jpg
properties-worksheet.jpg
en-US-worksheet.jpg

New Tobin Paradigm

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by New Tobin Paradigm » 2016-10-04, 23:23

One.. No one is going to be allowed to use an office suite to manage this. Get that through your head riiis.

Two.. What about the strings that need translated.. How are you going to make this collaborative. How are you able to cross reference a new or untranslated string with non-existent string or the translated one.

We read your thread.. We understand your viewpoint.. Are you going to head this up and create the infra needed for collaborative contributing, checking, auditing, as well as processing and packaging?
Last edited by New Tobin Paradigm on 2016-10-04, 23:28, edited 2 times in total.

User avatar
Sob__
Lunatic
Lunatic
Posts: 251
Joined: 2014-02-17, 01:12
Location: CZ

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Sob__ » 2016-10-05, 00:27

I tried something and it works. I almost dare to say it works well.

It's few php scripts. They read a "template" (that would be current v27 en-US langpack), compare what changed since previous v26 en-US langpack, get localized strings from v26 langpack, and produce semi-translated langpack with exactly the same structure as template. Every change is clearly marked, so that it can be easily found.

For example in .dtd files, newly added entity looks like this:

Code: Select all

<!-- PMTRANSLATE: added -->
<!ENTITY statusBar.label "Status Bar">
Changed entity looks like this:

Code: Select all

<!-- PMTRANSLATE: changed: Always show the button text / Vždy zobrazit text tlačítka => Always show the indicator text / ??? -->
<!ENTITY status4evar.download.label.force.label "Always show the indicator text">
The format of comment is:

Code: Select all

<!-- PMTRANSLATE: changed: <original en-US string> / <original localized string> => <new en-US string> / ??? -->
It's immediately clear what was changed. String in entity is changed to new en-US version. The goal is to produce langpack that does not break when original en-US version changes and localized version is not updated yet. It gets new/changed string from en-US version, and will continue to work just fine.
Translator can simply search for all files containing "PMTRANSLATE", open them in favourite editor, update translation, drop these helper comments in the process, and when finished, create patch against previous version. Opinions may differ, but it seems pretty straightforward to me. ;)

Current statistics for Czech language, with source for comparison being only v26 langpacks, are 7162 strings not changed at all, 1684 newly added strings (~1000 come from files with "devtools" in path, and it looks like devtools are doomed anyway), and 58 changed strings. Looks like a good starting point to me. First impression of generated langpack is very good. Translate few strings here and there and it's good enough for my mom to use.

Other languages tested are de, es-ES, fr, ja, ru and they all produce working langpacks (but not thoroughly tested).

Current state of this whole thing is proof of concept. Not exactly pretty inside, but so far it seems to work fine. If nothing else, I think it can be useful as helper tool for anyone wanting to start with v27 langpacks. In future, if it gets sufficiently polished, it could perhaps be used for automated updates of langpacks, when en-US source changes. Of course it's just a tool, it does not solve administrative matters.
Attachments
langupdater.alpha1.7z
(3.83 KiB) Downloaded 54 times

New Tobin Paradigm

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by New Tobin Paradigm » 2016-10-05, 00:45

Now that sounds like a constructive piece of the overall puzzle. Something that could be built upon for a true solution. Thanks!

GMforker

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by GMforker » 2016-10-05, 21:04

Sob__ wrote:Current statistics for Czech language, with source for comparison being only v26 langpacks, are 7162 strings not changed at all, 1684 newly added strings (~1000 come from files with "devtools" in path, and it looks like devtools are doomed anyway), and 58 changed strings.
Thank you for your interesting example! :-)

Btw, all new strings (cs) are translated (maybe - I don't have time to complete this - but I watch commits (Tycho) regularly). Of course, it is part of the version FF ESR 38 (cs) + fixed in new releases (but only for the desktop version; I'm not going to deal with the Android version, unfortunately).
It's just somewhere to put it...

User avatar
Sob__
Lunatic
Lunatic
Posts: 251
Joined: 2014-02-17, 01:12
Location: CZ

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Sob__ » 2016-10-06, 01:30

I added support for alternative translation source (FF38 langpack) and current stats are much better. Nothing changed for same and changed strings, but newly added (when comparing old and new en-US version) are down to 132 from original 1684. FF38 langpack provided 1552 translated strings. Alternative translation source also requires en-US FF38 langpack to make sure that English string did not change.

Original translation is preferred over FF38 translation. It would be tempting to prefer FF38 translation, because at least for Czech language it seems to be slightly better. Unfortunately, it can't be automated, because it would also make some unwanted changes, most notably it would change all occurences of "Pale Moon" to "Firefox". ;)
Attachments
langupdater.alpha2.7z
(4.14 KiB) Downloaded 54 times

GMforker

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by GMforker » 2016-10-06, 04:17

Sob__ wrote:I added support for alternative translation source (FF38 langpack)
8-)
Sob__ wrote:Original translation is preferred over FF38 translation. It would be tempting to prefer FF38 translation, because at least for Czech language it seems to be slightly better. Unfortunately, it can't be automated, because it would also make some unwanted changes, most notably it would change all occurences of "Pale Moon" to "Firefox". ;)
Yes, I have corrected these strings (only where it makes sense - so not... e.g.: https://github.com/MoonchildProductions ... erties#L11).

dark_moon

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by dark_moon » 2016-10-08, 10:02

Moonchild wrote:Attached a beta1 en-US pack.
Can we build with that a localization?
If yes, is editing the files in \browser\chrome\en-US\locale enough? Ah and edit \browser\defaults\preferences\firefox-l10n.js "pref("general.useragent.locale", "en-US");"

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Community involvement needed: Pale Moon v27+ and localization

Unread post by Moonchild » 2016-10-08, 16:34

*sigh*

Really, can we all stop trying to approach this from a practical translation implementation side for a moment, and look at the big picture that I started this thread for?
Yes, there are clear and straightforward ways to shape l10n for Pale Moon. What we need for this right now is someone to organize these efforts into a coherent whole. Actual translation, statistics of it, etc. is secondary to this issue. If there's nobody to guide this effort into proper procedures, paths and implementations, then piecemeal translation efforts will just sit unused for long whiles. I've tried to squeeze this into my schedule many times over, and it just isn't possible for me or any other of the core code developers to take on this task.

Once more: We need someone to take the reigns, to lead this effort, to set up a translation system that translators can use, and to take care of QA and packaging these language packs resulting from it. Once that is done, this same l10n lead will have to be the central point of communication for translators in case of questions, updates, etc.
dark_moon wrote:Can we build with that a localization?
If yes, is editing the files in \browser\chrome\en-US\locale enough? Ah and edit \browser\defaults\preferences\firefox-l10n.js "pref("general.useragent.locale", "en-US");"
You can, but be aware that it has already fallen behind because more en-US strings have been added to/changed in the browser since that pack was created. This is the kind of thing that needs to be streamlined with imports from the code base to keep foreign languages up-to-date with the state of the browser. Also, just \browser\chrome won't be enough - a proper localization has all strings translated.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked