dallaslu / gettext-commons Goto Github PK

I18n.trc returns comment when translation is not found instead of fallback
text of source locale

With gettext-commons 0.9.1.

I think the cache is broken with multiple concurrent clients.

 2008-03-10 12:11:52,512 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/].[jsp]]
El Servlet.service() para servlet jsp lanzó una excepción
java.util.ConcurrentModificationException
        at
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:449)
        at java.util.AbstractList$Itr.next(AbstractList.java:420)
        at org.xnap.commons.i18n.I18nCache.get(I18nCache.java:57)
        at org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:231)
        at org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:180)
        at org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:127)
        at org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:116)

What steps will reproduce the problem?
1. Run an ant target that contains this task:
<gettext-dist targetBundle="FOO" poDirectory="po" outputDirectory="build"/>

What is the expected output? What do you see instead?

I would like to have msgfmt invoked with the argument --verbose. I see no
way to specify this argument, or any custom arguments, for that matter.

What version of the product are you using? On what operating system?

gettext-ant-tasks-0.9.6 on Debian GNU/Linux 5.0 (Lenny)

Please provide any additional information below.

msgfmt --verbose would be useful for getting statistics.  The percentage
threshold is not enough for all projects.

I18nManager.setDefaultLocale(Locale newLocale) changes the locale for all
the I18n objects created via I18nFactory but I think it should also call
Locale.setDefault(newLocale) so that all new I18n objects would also come
with the newLocale.

Evaldas Juska

What steps will reproduce the problem?

Test with a simple i18nbuild.xml using the tutorial (adapted) on a Vista system

1. The task in the Ant build
-------------------------
<target name="generate-default-bundle" description="Generates a default bundle" 
depends="init.gettext">
  <gettext-generate-default targetBundle="messages" outputDirectory="${sourcesFilesDir}\po" potfile="po\messages.pot"/>
</target>

2. Run the task in the Ant build 


What is the expected output? What do you see instead?

Instead of the success of the build you get this failed error on the 
generate-default-bundle task :

Buildfile: C:\SVN\WLI\AMM_GDC-EJB\i18nBuild.xml
init.gettext:
generate-default-bundle:
[gettext-generate-default] Generating default po file: msginit --no-translator 
-i po\messages.pot -o C:\Users\ftrompat\AppData\Local\Temp\default24385.po
[gettext-generate-default] msgfmt --java2 -d C:\SVN\WLI\AMM_GDC-EJB\po -r 
messages C:\Users\ftrompat\AppData\Local\Temp\default24385.po
[gettext-generate-default] msgfmt returned 1

BUILD FAILED
C:\SVN\WLI\AMM_GDC-EJB\i18nBuild.xml:20: Build failed


What version of the product are you using? On what operating system?

Reproduced on a Vista system
with 
msgfmt command (version 0.14.4.1952)
gettext-ant-tasks-0.9.7
Bea WorkSpace Studio version 1.1

Please provide any additional information below.
I tested some workarounds on a cmd windows and I suspect the quotes in the 
file's or directory's path.
See the attached file to get more.

Regards

François

What steps will reproduce the problem?
1. Have Hebrew translations in a file po/he.po
2. Note that Java uses the old identifier iw_IL
3. Run <gettext-dist targetBundle="FOO" poDirectory="po"
outputDirectory="build"/>

What is the expected output? What do you see instead?

msgfmt ... -liw_IL po/he.po
Instead of this, gettext-merge is apparently invoking
msgfmt ... -lhe po/he.po

Please use labels and text to provide additional information.

I'm sorry if this is a documentation issue. (If it is, please clarify the
documentation.) I don't see a way to provide a list of mappings to msgfmt
that allows me to say that -liw_IL should be passed for he.po.

Compare:
i18n.tr("{0}", "replacement") -> "replacement"
i18n.tr("'{0}", "replacement") -> "'{0}"

Reported via email:

There is only version 0.9 available.

Addionally:
On http://xnap-commons.sourceforge.net/gettext-commons/maven.html you 
state that for maven 2, one should add:
<dependencies>
   <dependency>
     <groupId>org.xnap.commons</groupId>
     <artifactId>gettext-commons</artifactId>
     <version>0.9.1</version>
     <url>http://xnap-commons.sf.net/gettext-commons/</url>
   </dependency>
</dependencies>

this does not work, as version 0.9.1 is not available, and the <url> 
element is not allowed in the pom.xml. 

Class.getClassLoader() can return null in certain cases, which causes a 
NPE during lookup:

java.lang.NullPointerException
    at 
org.xnap.commons.i18n.I18nFactory.readFromPropertiesFile(I18nFactory.java:251)
    at 
org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:208)
    at 
org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:161)
    at 
org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:138)
    at 
org.xnap.commons.i18n.I18nFactory.getI18n(I18nFactory.java:146)
[...]

This way only translations will be bundled that are relatively complete.

What steps will reproduce the problem?
1. have a po file which contains a translation with non-matching \n at the
beginning of the string
i.e.:
msgid "\nWarning!\n"
msgstr "Achtung!"

2. using gnuwin32 gettext (0.14.4), gettext-ant-task version 0.9.7 (or earlier)
3. run the GettextDistTask Ant task 
The build will fail with the following message:
...build.xml:360: For input string: "C:\Dokumente"


This issue is caused by the GettextDiskTask's attempt to parse the output
of the command 'msgfmt --statistics path/to/the/pofile.po'
This usually looks like this:
"1244 translated messages, 43 fuzzy translations, 29 untranslated messages."

But in this case, the non-matching newlines generate an additional warning:
"...\lv.po:256: `msgid' and `msgstr' entries do not both begin with '\n'
16 translated messages, 4 fuzzy translations, 1296 untranslated messages."


What is the expected output? What do you see instead?

Ideally the parsing of the msgfmt output should be independent of such
warnings. It would be probably sufficient to parse the string starting at
the end, or taking into account the percentage number's labels.

Another way would be a less restrictive exception handling at this point to
continue the build for other languages.

I understand that the missing \n int the translation is actually a mistake
that should (and will) be fixed by the translators. But when maintaining a
large number of translations at different states of completion it is very
unfortunate that the whole build fails because of a warning produced for
one of the po files.


What version of the product are you using? On what operating system?
GnuWin32 gettext (0.14.4)
gettext-ant-task version 0.9.7

After maven clean
the first build  will fail because ${project.build.outputDirectory} may not 
exists when running DistMojo.

Solution before running msgfmtCmd
ensure that directory outputDirectory  is created.

e.g. call
outputDirectory.mkdirs() 

Add an option "createDefaultBundleFrom=keys.pot" to the gettext-dist target.

The error is in gettext-ant-tasks-0.9.1.jar
org.xnap.commons.ant.gettext.GettextDistTask.MsgCatCommandlineFactory.createComm
andline(File
file) method (line 163 in GettextDistTask.java). msgfmtCmd should be
replaced with msgcatCmd.

I think this is a simple copy paste mistake :)

I have attached the correct file for you.

Evaldas Juska

In I18n.java:

this could cause problems:
return translated == key ? text : translated;

it should be:
return translated.equals( key ) ? text : translated;


Regards, 
Rainer

Currently the Gettext Commons library fails if a ressource bundle can not
be located. While this is desirable in most situations it would be
convenient to allow a fall back to a default bundle. 

This should either be a configuration option when creating an I18n instance
or could be controlled in I18nFactory.

What steps will reproduce the problem?
1. .java files do not appear to have LGPL licence headers, as required by
the licence
2. Any kind of licence files are missing when building gettext-ant-tasks jar.

- Could you add licence information ?

What version of the product are you using? On what operating system?
- I am packaging gettext-ant-tasks release 0.9.6 for fedora, licence
informations are required.

The ant tasks don't work properly when the current working directory is
different from ant's base directory. This is a problem for at least those
of us who invoke ant via Eclipse, since Eclipse's cwd is its installation
directory.

Patch attached.

gettext msg context is available since version 0.17. Support it in gettext
commons.

When calling msgmerge I would like to pass it the --previous option to
allow translators to see what has changed since their last update.  Other
people might have other options they might want to pass onto the gettext
commands.

I needed to make the gettext plugin run on other extension than .java, so I 
added the sourceExtensions option to the GettextMojo.

Attached is a patch created on svn revision 214.

The sourceExtensions can be used as following in the pom configuration:
<build>
        <plugins>
            <plugin>
                <groupId>org.xnap.commons</groupId>
                <artifactId>maven-gettext-plugin</artifactId>
                <version>${gettext.version}</version>
                <configuration>
                    <targetBundle>com.upc.saleslayer.frontend.Messages</targetBundle>
                    <sourceDirectory>${project.build.sourceDirectory}/../</sourceDirectory>
                    <sourceLocale>${gettext.sourceLocale}</sourceLocale>
                    <sourceExtensions>
                        <param>java</param>
                        <param>ftl</param>
                    </sourceExtensions>
                    <keysFile>${gettext.keysFile}</keysFile>
                    <keywords>${gettext.keywords}</keywords>
                    <poDirectory>${gettext.poDirectory}</poDirectory>
                </configuration>
            </plugin>

When the sourceExtensions option is not configured in the POM, the java 
extension is added by default. When "java" is not set as extension, it will 
also be added to the sourceExtensions list by default. So "java" files are 
ALWAYS included in the gettext source list.

Hopefully this can be included in the next version of the maven-gettext-plugin.

Thanks

We run the build for application on different platforms and every developer has 
his own directory structure.

To avoid extra difs in translation file for different user it would be nice to 
use relative path in generated files.

Attached patch  tested on windows.

I18n.trc() will produce wrong (default) translations in the following
situation:

1. code Locale is "en"

2. used ResourceBundle is ROOT locale (i18n.Messages) - fallback from
non-existing i18n.Messages_en - but the I18n Locale language is "en"

3. I18n.trc() will check if text or translated comment should be used, and
for this uses the bundle's locale to determine if it's the same as the code
locale. 
As those are different (ROOT vs "en"), I18n.tr() will be invoked for the
default bundle with the comment instead of the text, which is not
translated but returned as is (ie. "boot (noun)" instead of "boot")


What version of the product are you using? On what operating system?

gettext-commons 0.9.2 with ant-tasks 0.9.3
 on WindowsXP with Eclipse 3.4.1.



Please provide any additional information below.

Problem here is, that the I18n that is returned by the factory has the
locale "en", but the actual ResourceBundle is the base, ROOT Locale. 

A possible fix, as far as I see, would be:

Ask for the I18n's Locale instead of the bundle's in I18n, line 488:

current:
return sourceCodeLocale.equals(getResources().getLocale()) ? text :
tr(comment);

fixed:
return sourceCodeLocale.equals(getLocale()) ? text : tr(comment);


I'm not sure if this would have other side effects. Maybe its also just me
using a wrong Locale set for the default Bundle?

What steps will reproduce the problem?

This Ant snippet will throw a StringIndexOutOfBoundsException in
gettext-ant-tasks 0.9.2 and 0.9.3:

    <gettext-extract keysFile="foo.pot" poDirectory="po"
                     keywords="-k -ktrc -ktr -kmarktr -ktrn:1,2 -ktrl">
      <fileset dir="." includes="foo*.java"/>
    </gettext-extract>

What is the expected output? What do you see instead?

It should start processing the files:

[gettext-extract] Executing: xgettext -c --from-code=utf-8 ...

Instead, it crashes, apparently while processing the file names:

BUILD FAILED
/home/marko/private/scratch/osm/josm/i18n/build.xml:42:
java.lang.StringIndexOutOfBoundsException: String index out of range: -1

What version of the product are you using? On what operating system?

0.9.2, 0.9.3 on Debian GNU/Linux

Please provide any additional information below.

Please consider applying the attached patch.

Allow multiple source directoris may be useful.

Usage example:

1)
I'm making i18n for GWT project.
This includes multiple generated java files.  I need to pars them as well.
Also in GWT there there may be java source files that are in resources 
directory. see super-source 
http://code.google.com/webtoolkit/doc/1.6/DevGuideOrganizingProjects.html
 'Overriding one package implementation with another'

2)
Another example if I want unpacks the project java dependencies to one 
'target/source-4i18n'  and then use xgettext on all this files.
N.B. for GWT all jar files must contain java sources.

Another alternative for this usage:

3) unpacks  sources for all dependencies to one folder using 
'maven-dependency-plugin'  before calling xgettext .
  The case is I need to include sources for current project and unpacked sources.

In my Maven projects, I configured the maven-gettext-plugin to
automatically run the dist goal in the compile phase. At the moment, every
time I run mvn compile, the gettext maven plugin compiles all existing PO
files using msgfmt, which takes a lot of time. The maven compiler plugin
instead compiles only those files that have been changed since the last
execution. It would be great if the maven gettext plugin also only compiled
those PO files whose compiled versions are out of date.

Also keep weak references in the cache so the I18n instances and 
underlying ResourceBundles can be freed up.

When I launch mvn org.xnap.commons:maven-gettext-plugin:dist, I've got this
answer :

What is the expected output? What do you see instead?
[INFO] Processing file-de.po
[WARNING] msgfmt: l'option « --no-location » n'est pas reconnue
[WARNING] Pour plus d'information, essayer « msgfmt --help »

More over, I don't understand why 'msgfmt' is called when I configure my
plugin's ouputFormat with "properties".


I use maven-gettext-plugin, version 1.1 and msgfmt version 0.15.

Thanks in advance,

Hi,  This is a request for you in future to release source tarballs.

This is to enable to Gentoo Linux to package gettext-commons.  Gentoo Linux
is a source based distribution, so it is very important that the source is
available for our users to download.  Please contact me on ali_bush @
gentoo dot org if you have any questions.

Hello,
is it possible to have the sources for gettext-ant-tasks as a separate
download? You already provide the source for gettext-commons, and it would
be useful to have the ant tasks too (in a separate tarball, maybe :))

Thanks!
David

unexpected output: 1284 translated messages.

Make sure all relevant information is ported to the new location.

Replace references to XNap Commons and add @since tags.

The exception thrown by I18nFactory.getI18n() when no resource bundle is 
found should contain more useful information to determine the cause of 
the failure.

What steps will reproduce the problem?
1. tried to extract some i18n-strings from a Mono-project with the ant-tasks
2. unfortunately it assumed that the lang is Java so nothing was extracted
3. therefore i patched the library to allow " language="--lang=C#" "
(--lang=java is the default value so existing projects don't break)

What is the expected output? What do you see instead?
attached is the diff-file

What version of the product are you using? On what operating system?
v0.9.7 on MacOS

Change look up to also check default package for ResourceBundle in 
I18nFactory.

When you use Maven POM inheritance and have a lot of sub-modules in a Maven
project, you usually do not want to configure the compiling of the PO files
in each sub-module separately, but configure it in a super POM that is
inherited in all modules. Unfortunately, if you have some modules that
contain PO files and some that don’t, this becomes hard as the
maven-gettext-plugin will fail to execute the dist task if the configured
poDirectory does not exist.

To facilitate this kind of project structure, it would be very useful to
have a parameter that defines whether to ignore the missing of the poDirectory.

What steps will reproduce the problem?
1. cd gettext-commons-0.9.6
2. javadoc src/java/org/xnap/commons/i18n/I18n.java -d javadoc
3.

What is the expected output? 
There are 4 warnings about a missing closing '}' character. The attached
patch fixes these.

What do you see instead?

Creating destination directory: "javadoc/"
Loading source file src/java/org/xnap/commons/i18n/I18n.java...
Constructing Javadoc information...
Standard Doclet version 1.6.0_0
Building tree for all the packages and classes...
Generating javadoc/org/xnap/commons/i18n//I18n.html...
src/java/org/xnap/commons/i18n/I18n.java:552: warning - Missing closing '}'
character for inline tag: "{@link #trnc(String, String, String, long,
Object[]) passing <code>obj</code>
 arguments as an array."
src/java/org/xnap/commons/i18n/I18n.java:563: warning - Missing closing '}'
character for inline tag: "{@link #trnc(String, String, String, long,
Object[]) passing <code>obj1</code> and <code>obj2</code> 
 arguments as an array."
src/java/org/xnap/commons/i18n/I18n.java:574: warning - Missing closing '}'
character for inline tag: "{@link #trnc(String, String, String, long,
Object[]) passing <code>obj1</code>, <code>obj2</code> and <code>obj3</code>
 arguments as an array."
src/java/org/xnap/commons/i18n/I18n.java:585: warning - Missing closing '}'
character for inline tag: "{@link #trnc(String, String, String, long,
Object[]) passing <code>obj1</code>, <code>obj2</code>, <code>obj3</code>
and <code>obj4</code>
 arguments as an array."
src/java/org/xnap/commons/i18n/I18n.java:515: warning - @param argument
"text" is not a parameter name.
src/java/org/xnap/commons/i18n/I18n.java:541: warning - @param argument
"text" is not a parameter name.
Generating javadoc/org/xnap/commons/i18n//package-frame.html...
Generating javadoc/org/xnap/commons/i18n//package-summary.html...
Generating javadoc/org/xnap/commons/i18n//package-tree.html...
Generating javadoc/constant-values.html...
Building index for all the packages and classes...
Generating javadoc/overview-tree.html...
Generating javadoc/index-all.html...
Generating javadoc/deprecated-list.html...
Building index for all classes...
Generating javadoc/allclasses-frame.html...
Generating javadoc/allclasses-noframe.html...
Generating javadoc/index.html...
Generating javadoc/help-doc.html...
Generating javadoc/stylesheet.css...
6 warnings

See issue #1.

The ant task should support generating message bundles from locales that 
have a variant, e.g. sr@Latn.

If I18nFactory.getI18n() is invoked with a baseName the lookup should not 
search for i18n.properties files.

What steps will reproduce the problem?
1. Use I18n.trc(context, text) with arguments like trc("chat (noun)", "chat")
2. The translation to another language will attempt to look up "chat
(noun)\u0004chat" instead of "chat (noun)"


What is the expected output? What do you see instead?

The method comment and the way gettext extracts strings suggest that only
context will be translated, not a concatenation of context and text:

 * @return <code>text</code> if the locale of the underlying resource
 *         bundle equals the source code locale, the translation of
 *         <code>comment</code> otherwise.


The testcase (I18nTest) tests for a different behavior, where the context
is only a disambiguation string without containing the text ("(noun)"
instead of "chat (noun)". 
This will lead to problems with the string extraction, as the text to
translate will show up as "(noun)" in the catalog.po file, thus making
translations difficult.


What version of the product are you using? On what operating system?
gettext-commons 0.9.6

Please provide any additional information below.

Proposed fix:

public final String trc(String context, String text)
{
    if (sourceCodeLocale.equals(getResources().getLocale())) { 
        return text;
    } else {
        String translated = tr(context);
        // if no translation was found return text in source locale
        return translated == key ? text : translated;
    }
}

Same holds for the method
I18n.trnc(String context, String singularText, String pluralText, long n)

And log any error outputs.

The default should match the Maven plug-in / gettext-commons API:

-ktrc -ktr -kmarktr -ktrn:1,2

This makes the task more flexible.

What steps will reproduce the problem?
1. add i18n.trc("asdf", "jklö") somewhere
2. change the locale to something else but default
3. text from default language will be taken instead of translated text

Or is there something I don't know about trc?

I want to use classes that includes i18n  in JUnit tests.

I want the tests to to execute in any environment:
e.g. I18nFactory.getI18n(getClass(), I18nFactory.FALLBACK);   should work.

problem is that I can't use functions with 'java.util.Locale' because of the 
GWT limitations.

I’ve been using the maven plugin for some time and found it quite
useful. The point is that now I would need to pass a couple of
parameters to the xgettext application that are not included in the
configuration parameters, so I wonder if you’re thinking on adding any
way of doing that.

I modified the source code to fit my current needs but I find that
adding one more parameter that allows to pass more options to the GNU
applications would be a nice feature.

Something like the following would be great

<plugin>
   <groupId>org.xnap.commons</groupId>
   <artifactId>maven-gettext-plugin</artifactId>
   <version>1.2.0</version>
   <executions>
        <execution>
           <id>distribute-properties-files</id>
           <configuration>
                <poDirectory>src/main/po</poDirectory>
                <targetBundle>com.kinamik.vault.Messages</
targetBundle>
                <outputFormat>properties</outputFormat>
                <options>--no-location –add-comments=///</options>
           </configuration>
           <goals>
              <goal>dist</goal>
           </goals>
        </execution>
   </executions>
</plugin>

The PO file should not contain absolute pathes for the sources.

The current repositiory version of the GettextDistTask will halt the build
when the msgfmt call returns a result other than 0.

Warnings which don't have to break the build will return 1.


What steps will reproduce the problem?
1. have a po file that causes a warning when processed by GettextDistTask,
like a translation missing a leading "\n"
2. run GettextDistTask

What is the expected output? What do you see instead?

Because of the warning, msgfmt seems to return a value of 1. This causes
AbstractGettextTask to throw an exception which causes the build to halt.

This is not necessary, as not even the language that causes the message has
to be necessarily erroneous. 


What version of the product are you using? On what operating system?
gettext-ant-taks, prerelease 0.9.8 (revision 206)


Please provide any additional information below.

See also issue "msgfmt warning output breaks GettextDistTask", which
already solved parts of this problem for the parsing of the msgfmt command
line output.