Localization server help needed

The l10n of Pale Moon. Rawr.
squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 15:01

Okay, done :)

tree2flat.sh:

Code: Select all

#!/bin/bash

# Flattens a predetermined tree directory structure.

# The MIT License (MIT)
# Copyright (C) 2015 Squarefractal <squarefractal at gmail {.} com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

echoerr () {
	echo "ERROR: $1" >&2
}

print_usage () {
	echo "Usage: $0 <input_dir> <output_dir>"
	echo "Flattens a predetermined tree directory structure."
	exit 0
}

if [[ -z $1 ]] || [[ -z $2 ]] || [[ $1 == "--help" ]]; then
	print_usage
fi

input_dir="$(readlink -e "$1")"
output_dir="$(readlink -e "$2")"

if [[ ! -d $input_dir ]]; then
	echoerr "Invalid path to localization file"
	exit 1
fi

if [[ ! -d $output_dir ]]; then
	echoerr "Invalid extraction directory"
	exit 1
fi

cd "$input_dir"

find . -type f -name '*.po' | sed -r 's|^./||g' | while read filepath; do
	cp -v "$filepath" "$output_dir/$(echo "$filepath" | sed 's|/|!|g')"
done

Code: Select all

#!/bin/bash

# Converts a predetermined flattened directory structure to a tree.

# The MIT License (MIT)
# Copyright (C) 2015 Squarefractal <squarefractal at gmail {.} com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

echoerr () {
	echo "ERROR: $1" >&2
}

print_usage () {
	echo "Usage: $0 <input_dir> <output_dir>"
	echo "Converts a predetermined flattened directory structure to a tree."
	exit 0
}

if [[ -z $1 ]] || [[ -z $2 ]] || [[ $1 == "--help" ]]; then
	print_usage
fi

input_dir="$(readlink -e "$1")"
output_dir="$(readlink -e "$2")"

if [[ ! -d "$input_dir" ]]; then
	echoerr "Invalid input directory."
	exit 1
fi

if [[ ! -d "$output_dir" ]]; then
	echoerr "Invalid output directory."
	exit 1
fi

cd "$input_dir"

find . -type f -name '*.po' | sed -r 's|^./||g' | while read filepath; do
	dest="$(echo "$filepath" | sed -r 's|!|/|g;s|(.*)/.*|\1|g')"
	fname="$(echo "$filepath" | sed -r 's|.*!||g')"
	mkdir -vp "$output_dir/$dest"
	cp -v "$filepath" "$output_dir/$dest/$fname"
done
Off-topic:
Any developer should really learn bash: it helps a lot!
And batch is not a language, please.
Last edited by squarefractal on 2015-03-24, 18:54, edited 1 time in total.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 15:26

Awesome! Thanks for your help, I'll see if this works and if it needs some polish or shave I'll do it myself, but looks good at first glance.
Off-topic:
squarefractal wrote:Any developer should really learn bash: it helps a lot!
And batch is not a language, please.
Batch file programming is just as versatile as shellscripts. We've come a long way since DOS.
And if you need even more there's powershell ;)
Same kind of small, plain scripting, just a different OS.

And bash is not a language, please. ;-) 8-)
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 15:39

Problem: your script relies on several utilities, of which one I don't have available in my mingw shell: readlink
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 16:45

Code: Select all

mingw-get install msys-coreutils
Also check whether the msys/1.0/ binary paths are in %PATH%.

Lucio Chiappetti
Astronaut
Astronaut
Posts: 654
Joined: 2014-09-01, 15:11
Location: Milan Italy

Re: Localization server help needed

Unread post by Lucio Chiappetti » 2015-03-24, 17:22

Let's see if I've understood ... (1) you can work on Linux ? (2) you do not want to copy the files somewhere (in which case a combination of rsync and tar might do it) but to rename them.

If you can work in csh ... unfortunately you have the worst case because the path separator (slash /) and the exclamation mark (!) are special character and one cannot use variable substitution as $i:as/A/B/ with $i:as/\//\/ so one has to use tr and the octal representation of slash

- cd to the root of your tree
- store the list of po files in a shell variable
and for each name generate a new variable replacing slash with !
then construct the command to copy the file to DESTINATION
(the example is willingly not operational, it displays the command but does not execute it; remove word "echo" to execute)

Code: Select all

set list = `find . -name '*po' -print`
  foreach i ($list)
    set a = `echo $i | tr '\057' '!'`
    echo cp $i DESTINATION/$a
  end
The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. (G.B. Shaw)

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 17:29

I guess my msys installation sucks because I couldn't "get" anything. I'm installing a fresh MingW environment to pull the needed tools out of.

Why is readlink needed anyway?


@Lucio: I work on Windows. I do not have access to csh.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 17:55

I initially had used readlink assuming a Linux system (well, that's the impression I formed from your post) and using readlink helps me to cut down on the actual validation I have to do (if its a symlink, resolve it recursively, and check if its a directory or file, if it's broken or not, or maybe the file doesn't exist at all).

If even mingw-get is not available on your system, I guess you can just set xxx_dir=$(readlink -e "$n") to xxx_dir="$n", at the risk of breaking the script in case an actual symlink is there... I'm not sure about how Mingw behaves in that aspect.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 17:59

More problems, squarefractal... did you test your script?

1. it won't create output folders per language and just print an error. That's not very useful.
2. when an output folder is created manually, it will not work, either:

Code: Select all

./tree2flat.sh: cannot make pipe for process substitution: Function not implemen
ted
./tree2flat.sh: cannot make pipe for process substitution: Function not implemen
ted
./tree2flat.sh: line 57: <(find . -type f -name '*.po'): ambiguous redirect
One of these for each output folder.

Looks like it chokes on ... something.

I don't use symlinks, so this whole mess is not even needed. (why the hell would I symlink files I need to work with, anyway?... symlinks do have a place, but this isn't it :P )
I work in Windows, you should know this ;) I just have access to msys since that is the build environment for Mozilla code as well.

I'll rewrite the folder assignments and see if that works any better.

EDIT: still no go on the above error. The script I have now is this:

Code: Select all

#!/bin/bash

# Flattens a predetermined tree directory structure.

# The MIT License (MIT)
# Copyright (C) 2015 Squarefractal <squarefractal at gmail {.} com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

echoerr () {
   echo "ERROR: $1" >&2
}

print_usage () {
   echo "Usage: $0 <input_dir> <output_dir>"
   echo "Flattens a predetermined tree directory structure."
   exit 0
}

if [[ -z $1 ]] || [[ -z $2 ]] || [[ $1 == "--help" ]]; then
   print_usage
fi

input_dir="$1"
output_dir="$2"

if [[ ! -d $input_dir ]]; then
   echoerr "Invalid path to localization file"
   exit 1
fi

if [[ ! -d $output_dir ]]; then
   mkdir $output_dir
fi

cd "$input_dir"

while read filepath; do
   cp -v "$filepath" "$output_dir/$(echo "$filepath" | sed -r 's|^./||g;s|/|!|g')"
done < <(find . -type f -name '*.po')
It seems to choke on your cp magic line, and I really don't know enough about shellscripting to understand exactly what you're trying to do or why it chokes.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-24, 19:14

So you have an old version of bash doesn't implement process substitution... I've edited the scripts above to use pipes instead but you still need to manually change that readlink assignment (I won't support a broken system.)

If you run into further problems, just do the stuff in a Linux VM for now and install MSYS2 <http://sf.net/p/msys2> which actually updates things. (Looks like they still use some old version of bash, and haven't updated in years.)

BTW, all scripts have been tested before being posted here

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-03-24, 20:57

Yeah I understand the frustration. I also ask you understand mine: None of this would be needed if stuff actually worked the way it was supposed to, and I can't keep reinventing the wheel myself either. ;)
As for a "broken system" -- readlink is just not needed because symlinks are not something to take into account. No need to think of contingencies or build in safeguards for things that will never happen. I had no issue pulling readlink in from a fresh install and you should note that this fresh install, the current MinGW on SF, uses a bash shell that does not do what you tried either. So, you can call MinGW broken then. Next time I ask for help I'll make sure to clearly specify my OS and what I have available. How's that? 8-)

Using a Linux VM is simply not something I can fit into my daily routine beyond using it as a build slave. Pumping files back and forth between a VM and a host is also a PITA and often subject to a host of its own problems, so I'd rather avoid that.

As an aside, I almost got it to work with batch files as well (using variable substitution) but dealing with Windows being finnicky as to when it expands everything to a full path up to the drive level (which wouldn't work) had me short on time to figure something out. Hence the request for help, save having to write something from scratch because I know *nix-alike scripts are better at juggling mass file processing.

EDIT: Needs more work. cding into the dir makes the copy fail because you're no longer in the same directory level. I'll fix it.

EDIT2: making sure to generate a proper call script with top-level paths fixed the problem, and it's currently slowly chugging away at copying files. Thanks squarefractal! Playing the pipes worked ;)
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-03-25, 04:30

Pumping files back and forth between a VM and a host is also a PITA
Come on, typing "sudo mount -t vboxsf sharedname /mnt/shared" is not a PITA (assuming Virtualbox), whichever way you look at it.
Hence the request for help, save having to write something from scratch because I know *nix-alike scripts are better at juggling mass file processing.
On Windows, there is no such clear advantage since process spawning is extremely slow and I have seen scripts that complete their job in minutes under *nix take an hour on Windows.
the current MinGW on SF, uses a bash shell that does not do what you tried either.
EDIT: Needs more work. cding into the dir makes the copy fail because you're no longer in the same directory level. I'll fix it.
EDIT2: making sure to generate a proper call script with top-level paths
Can I get more details so I may fix the script (i.e. with an example of the invocation)? Because here it works with invocations like "../../tree2flat.sh . ../flat".
Also, are even pipes not working on the MinGW shell?

Lucio Chiappetti
Astronaut
Astronaut
Posts: 654
Joined: 2014-09-01, 15:11
Location: Milan Italy

Re: Localization server help needed

Unread post by Lucio Chiappetti » 2015-03-25, 11:04

Moonchild wrote:Using a Linux VM is simply not something I can fit into my daily routine beyond using it as a build slave. Pumping files back and forth between a VM and a host
I confess I never tried a config with a "real" Windows system hosting a "virtual" Linux. I have the other way round (real Linux hosting virtual Windows XP) and (with VirtualBox) I see the Linux disks as Windows drives without any effort(no need even to install SAMBA which is something I did in a far past). My main use of the XP VM is to run an utility for visual diff of PDF files (which I haven't found for Linux).
The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. (G.B. Shaw)

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-04-23, 08:15

Looks like I'm still at square one. :/

OneSky, despite all the attempts at getting it to work simply doesn't accept the files, doesn't like to work with files in folders, doesn't work well with the converted PO files and doesn't support the formats natively. Even going through all the effort converting and renaming files, it's bust, and their support department is NOT responsive. So, scratch that.

I'll have to give bitnami pootle another go, maybe I'll have more luck if I install it on a different OS.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-04-23, 15:02

What's the problem with a normal git repo that a contributors can clone, commit changes and create a PR?

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-04-23, 16:05

squarefractal wrote:What's the problem with a normal git repo that a contributors can clone, commit changes and create a PR?
The problem is that there will be no easy way to see if strings have been added, which translations are incomplete and which aren't, no easy side-by-side comparison of strings (unless you open 2 documents side by side to translate...), etc.

The repo is already available: https://github.com/MoonchildProductions/pale-moon-localization

But as you can imagine, it needs some sort of management interface to keep overview.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-04-23, 16:09

I've managed to get the pootle VM up and running -- but running into the next brick wall: there is no actual documentation provided by translatehouse for the administration/project management side of things -- I have no clue how to:
  • Set up the initial language as a template to be used for other languages
  • Import existing translations
If anyone is familiar with this software, please help. Translatehouse's documentation is a few paragraphs just outlining what it does, has no howtos or examples, and doesn't even touch on importing existing translations into it. Thankfully it is smart enough to automatically parse a language pack when uploaded, but I'm running aground for anything else to properly set it up.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

squarefractal

Re: Localization server help needed

Unread post by squarefractal » 2015-04-23, 16:41

While I have not used Pootle at all, have you looked at this?
* "Import existing translations" : http://docs.translatehouse.org/projects ... fline.html

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-04-23, 18:03

Yes I did, and it didn't help - it's for importing files that were translated off-line (exported from Pootle, translated, then uploaded again to Pootle, as opposed to translating on-line), not for setting up a project, unless I completely missed something.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

KNTRO

Re: Localization server help needed

Unread post by KNTRO » 2015-04-24, 05:10

Have you gave oTranCe a chance, MC?

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35478
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Localization server help needed

Unread post by Moonchild » 2015-04-24, 09:17

KNTRO wrote:Have you gave oTranCe a chance, MC?
I had a look at it just now; it doesn't seem to be suited for Pale Moon (no import options for the used file formats, no support for many files).
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked