Localization server help needed

The l10n of Pale Moon. Rawr.
User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Localization server help needed

Unread post by Moonchild » 2015-03-21, 12:05

OK folks,

I've tried very hard, over the past few months, to get a localization server setup for Pale Moon's language packs. And unfortunately it's just been a big failure.

The best match for what we need seems to be Pootle, since it natively supports Mozilla's localization formats and is fully web based, but it is a python virtualenv mess that I simply can't get to work no matter how hard I try (ranging from not being able to use MySQL as a db back-end which is really needed, to a simple test setup listening on the correct port but being unable to connect to it...).
I have to admit defeat here -- I just can't do it, I can't get it to work. Python must hate me, or something.

Commercial alternatives offered are out of Pale Moon's budget range, the designer of Pootle (Translationhouse) doesn't offer hosted server instances, and other packages I looked at simply don't support the Mozilla file formats or require a ridiculous amount of manual pre- and post-processing.

I have a VPS I got for this purpose as a dedicated server for it. If anyone is willing to give this a shot I'll be more than happy to assign a maintainer to use the VPS with full (su) access (assuming your integrity is OK and you can be trusted not to abuse your power and access) using any preferred common Linux OS on it to get pootle set up (or any other package that would work well, really -- at this point I just want SOME form of l10n setup that I can use). I can even pay you some money for setting it up and maintaining it; it doesn't have to be charity work.

Hoping for more than just crickets,
MC
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Sob__
Lunatic
Lunatic
Posts: 251
Joined: 2014-02-17, 01:12
Location: CZ

Re: Localization server help needed

Unread post by Sob__ » 2015-03-21, 23:37

Just in case you missed it, what about https://bitnami.com/stack/pootle? I tried their installer on CentOS and server was running in five minutes (as far as I can tell, there are no obvious errors). I'd be a little worried about upgrades, as it seems that it will be more manual work. But the quick start is impressive.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-22, 00:55

Sob__ wrote:Just in case you missed it, what about https://bitnami.com/stack/pootle? I tried their installer on CentOS and server was running in five minutes (as far as I can tell, there are no obvious errors). I'd be a little worried about upgrades, as it seems that it will be more manual work. But the quick start is impressive.
Completely unfamiliar with bitnami, never heard of it before now -- and it did not come up in any of my searches.
One snag: the installer seems to be GUI -- I won't have GUI access on the server since I don't have an X desktop when administering remotely. Any idea how to use it in that situation, because they seem to assume everyone has local access with a graphical desktop on Linux servers... :P
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Sob__
Lunatic
Lunatic
Posts: 251
Joined: 2014-02-17, 01:12
Location: CZ

Re: Localization server help needed

Unread post by Sob__ » 2015-03-22, 02:53

I don't have GUI in my test VM either. At most, perhaps some X library as part of crazy dependencies of something else, but that would be it. Anyway, the installer (bitnami-pootle-2.5.1-5-linux-x64-installer.run) run happily in command line and it did not complain or ask for anything else.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-22, 12:00

Sob__ wrote:the installer (bitnami-pootle-2.5.1-5-linux-x64-installer.run) run happily in command line and it did not complain or ask for anything else.
Heh, all right. I'll give it a shot then - thanks for the pointer!
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-22, 14:15

Installed, got it up and running, but it seems like the actual Pootle package is not usable that this stack was built from.

It's labeled as a Demo install of pootle, and that is exactly what it acts as. :/
Or, maybe I'm just not able to understand the way one is supposed to work with pootle? I followed what scarce information there was to be found on-line, about converting moz files to po, creating templates, uploading files, etc.
it keeps reverting text to "demo" and the default description for the server no matter how often I change it...

Too much of a headache.

I think I'll just have to give up on trying to provide language packs this way, and find a different, possibly off-line, translation solution for this problem.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-22, 16:07

Moonchild wrote:it keeps reverting text to "demo" and the default description for the server no matter how often I change it...
Edit apps/pootle/etc/pootle/pootle-settings.conf to change the defaults. It works for me, see the screenshot below.
(In any case you should be reading the config file, since it contains important hints about how you should set your server up.)
Attachments
pootle-debian.png

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-22, 23:55

Unfortunately it just doesn't want to cooperate. I got it to accept the changes and display them... up to a random point in time where it would simply revert to "demo" and lose my entered title and description and settings.

I managed to get as far as manually converting the language files to a format it can understand, and managed to get it uploaded after a few tries, but it seems less work and having more overview to just have people edit files manually than to work with this mess :P
It keeps losing settings randomly, for starters (seems to be my bane for anything python-related: seems threads die randomly and consequently take any changes in them with them into the void). It doesn't allow me to import existing translations in an understandable fashion (upload PO files and it marks everything as "untranslated"), it doesn't understand I already have a source language and templates, and I got so frustrated with it all i just wiped the VPS to MAYBE start again with something else if i can find something better.

At least i got a handle on how many words the language packs are: a little over 33,000. :!:
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 13:07

It looks like I can use OneSky after preprocessing stuff with the translation toolkit (I got that covered), but the problem is that they don't support folder structures.

Since my advanced batch programming is very rusty, I could use some help. I need a simple way to automatically copy files, recursively, to a flat file structure, renaming the files in the process, in both directions.

What I would need is something that will process files as follows:

Input: a folder tree as in the attached zip

Output: all .po files (nothing else) in a single, flat folder, where file names have paths prepended with separators
e.g.: browser\chrome\ach\locale\browser\file.po becomes browser!chrome!ach!locale!browser!file.po

I'd also need something to copy these files back to folders later on, to replace previous versions of the files.

I don't really care in what way it's done, I can use .cmd, .sh, native windows binaries, even .py or .pl if needed.
Attachments
ach.zip
(423.22 KiB) Downloaded 65 times
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 14:45

{{in reply to posted, but removed, script}}

Thanks for the script! That was quick.

However, you may have misunderstood what I was asking: It would take a folder as input, not a zip, but "laid out as seen in the zip"
So I guess the whole zip extraction thing should not be done, but the copy operation should be selective on *.po, instead.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 15:01

Okay, done :)

tree2flat.sh:

Code: Select all

#!/bin/bash

# Flattens a predetermined tree directory structure.

# The MIT License (MIT)
# Copyright (C) 2015 Squarefractal <squarefractal at gmail {.} com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

echoerr () {
	echo "ERROR: $1" >&2
}

print_usage () {
	echo "Usage: $0 <input_dir> <output_dir>"
	echo "Flattens a predetermined tree directory structure."
	exit 0
}

if [[ -z $1 ]] || [[ -z $2 ]] || [[ $1 == "--help" ]]; then
	print_usage
fi

input_dir="$(readlink -e "$1")"
output_dir="$(readlink -e "$2")"

if [[ ! -d $input_dir ]]; then
	echoerr "Invalid path to localization file"
	exit 1
fi

if [[ ! -d $output_dir ]]; then
	echoerr "Invalid extraction directory"
	exit 1
fi

cd "$input_dir"

find . -type f -name '*.po' | sed -r 's|^./||g' | while read filepath; do
	cp -v "$filepath" "$output_dir/$(echo "$filepath" | sed 's|/|!|g')"
done

Code: Select all

#!/bin/bash

# Converts a predetermined flattened directory structure to a tree.

# The MIT License (MIT)
# Copyright (C) 2015 Squarefractal <squarefractal at gmail {.} com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

echoerr () {
	echo "ERROR: $1" >&2
}

print_usage () {
	echo "Usage: $0 <input_dir> <output_dir>"
	echo "Converts a predetermined flattened directory structure to a tree."
	exit 0
}

if [[ -z $1 ]] || [[ -z $2 ]] || [[ $1 == "--help" ]]; then
	print_usage
fi

input_dir="$(readlink -e "$1")"
output_dir="$(readlink -e "$2")"

if [[ ! -d "$input_dir" ]]; then
	echoerr "Invalid input directory."
	exit 1
fi

if [[ ! -d "$output_dir" ]]; then
	echoerr "Invalid output directory."
	exit 1
fi

cd "$input_dir"

find . -type f -name '*.po' | sed -r 's|^./||g' | while read filepath; do
	dest="$(echo "$filepath" | sed -r 's|!|/|g;s|(.*)/.*|\1|g')"
	fname="$(echo "$filepath" | sed -r 's|.*!||g')"
	mkdir -vp "$output_dir/$dest"
	cp -v "$filepath" "$output_dir/$dest/$fname"
done
Off-topic:
Any developer should really learn bash: it helps a lot!
And batch is not a language, please.
Last edited by squarefractal on 2015-03-24, 18:54, edited 1 time in total.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 15:26

Awesome! Thanks for your help, I'll see if this works and if it needs some polish or shave I'll do it myself, but looks good at first glance.
Off-topic:
squarefractal wrote:Any developer should really learn bash: it helps a lot!
And batch is not a language, please.
Batch file programming is just as versatile as shellscripts. We've come a long way since DOS.
And if you need even more there's powershell ;)
Same kind of small, plain scripting, just a different OS.

And bash is not a language, please. ;-) 8-)
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 15:39

Problem: your script relies on several utilities, of which one I don't have available in my mingw shell: readlink
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 16:45

Code: Select all

mingw-get install msys-coreutils
Also check whether the msys/1.0/ binary paths are in %PATH%.

Lucio Chiappetti
Astronaut
Astronaut
Posts: 654
Joined: 2014-09-01, 15:11
Location: Milan Italy

Re: Localization server help needed

Unread post by Lucio Chiappetti » 2015-03-24, 17:22

Let's see if I've understood ... (1) you can work on Linux ? (2) you do not want to copy the files somewhere (in which case a combination of rsync and tar might do it) but to rename them.

If you can work in csh ... unfortunately you have the worst case because the path separator (slash /) and the exclamation mark (!) are special character and one cannot use variable substitution as $i:as/A/B/ with $i:as/\//\/ so one has to use tr and the octal representation of slash

- cd to the root of your tree
- store the list of po files in a shell variable
and for each name generate a new variable replacing slash with !
then construct the command to copy the file to DESTINATION
(the example is willingly not operational, it displays the command but does not execute it; remove word "echo" to execute)

Code: Select all

set list = `find . -name '*po' -print`
  foreach i ($list)
    set a = `echo $i | tr '\057' '!'`
    echo cp $i DESTINATION/$a
  end
The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. (G.B. Shaw)

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 17:29

I guess my msys installation sucks because I couldn't "get" anything. I'm installing a fresh MingW environment to pull the needed tools out of.

Why is readlink needed anyway?


@Lucio: I work on Windows. I do not have access to csh.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 17:55

I initially had used readlink assuming a Linux system (well, that's the impression I formed from your post) and using readlink helps me to cut down on the actual validation I have to do (if its a symlink, resolve it recursively, and check if its a directory or file, if it's broken or not, or maybe the file doesn't exist at all).

If even mingw-get is not available on your system, I guess you can just set xxx_dir=$(readlink -e "$n") to xxx_dir="$n", at the risk of breaking the script in case an actual symlink is there... I'm not sure about how Mingw behaves in that aspect.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 17:59

More problems, squarefractal... did you test your script?

1. it won't create output folders per language and just print an error. That's not very useful.
2. when an output folder is created manually, it will not work, either:

Code: Select all

./tree2flat.sh: cannot make pipe for process substitution: Function not implemen
ted
./tree2flat.sh: cannot make pipe for process substitution: Function not implemen
ted
./tree2flat.sh: line 57: <(find . -type f -name '*.po'): ambiguous redirect
One of these for each output folder.

Looks like it chokes on ... something.

I don't use symlinks, so this whole mess is not even needed. (why the hell would I symlink files I need to work with, anyway?... symlinks do have a place, but this isn't it :P )
I work in Windows, you should know this ;) I just have access to msys since that is the build environment for Mozilla code as well.

I'll rewrite the folder assignments and see if that works any better.

EDIT: still no go on the above error. The script I have now is this:

Code: Select all

#!/bin/bash

# Flattens a predetermined tree directory structure.

# The MIT License (MIT)
# Copyright (C) 2015 Squarefractal <squarefractal at gmail {.} com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

echoerr () {
   echo "ERROR: $1" >&2
}

print_usage () {
   echo "Usage: $0 <input_dir> <output_dir>"
   echo "Flattens a predetermined tree directory structure."
   exit 0
}

if [[ -z $1 ]] || [[ -z $2 ]] || [[ $1 == "--help" ]]; then
   print_usage
fi

input_dir="$1"
output_dir="$2"

if [[ ! -d $input_dir ]]; then
   echoerr "Invalid path to localization file"
   exit 1
fi

if [[ ! -d $output_dir ]]; then
   mkdir $output_dir
fi

cd "$input_dir"

while read filepath; do
   cp -v "$filepath" "$output_dir/$(echo "$filepath" | sed -r 's|^./||g;s|/|!|g')"
done < <(find . -type f -name '*.po')
It seems to choke on your cp magic line, and I really don't know enough about shellscripting to understand exactly what you're trying to do or why it chokes.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 19:14

So you have an old version of bash doesn't implement process substitution... I've edited the scripts above to use pipes instead but you still need to manually change that readlink assignment (I won't support a broken system.)

If you run into further problems, just do the stuff in a Linux VM for now and install MSYS2 <http://sf.net/p/msys2> which actually updates things. (Looks like they still use some old version of bash, and haven't updated in years.)

BTW, all scripts have been tested before being posted here

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 20:57

Yeah I understand the frustration. I also ask you understand mine: None of this would be needed if stuff actually worked the way it was supposed to, and I can't keep reinventing the wheel myself either. ;)
As for a "broken system" -- readlink is just not needed because symlinks are not something to take into account. No need to think of contingencies or build in safeguards for things that will never happen. I had no issue pulling readlink in from a fresh install and you should note that this fresh install, the current MinGW on SF, uses a bash shell that does not do what you tried either. So, you can call MinGW broken then. Next time I ask for help I'll make sure to clearly specify my OS and what I have available. How's that? 8-)

Using a Linux VM is simply not something I can fit into my daily routine beyond using it as a build slave. Pumping files back and forth between a VM and a host is also a PITA and often subject to a host of its own problems, so I'd rather avoid that.

As an aside, I almost got it to work with batch files as well (using variable substitution) but dealing with Windows being finnicky as to when it expands everything to a full path up to the drive level (which wouldn't work) had me short on time to figure something out. Hence the request for help, save having to write something from scratch because I know *nix-alike scripts are better at juggling mass file processing.

EDIT: Needs more work. cding into the dir makes the copy fail because you're no longer in the same directory level. I'll fix it.

EDIT2: making sure to generate a proper call script with top-level paths fixed the problem, and it's currently slowly chugging away at copying files. Thanks squarefractal! Playing the pipes worked ;)
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked