[GR-52826] Non-ASCII characters in command line arguments are replaced by U+FFFD in Windows (native-image) about graal HOT 10 OPEN

Very nice! You've found THE dialog. I confirm that if UTF-8 is activated there, both classic launching and native-image build work consistently and correctly.

However, the UTF-8 option is not the default and the majority of users don't know about this dialog. This dialog is a legacy of pre-Unicode days. As you see, it's intended for "non-Unicode programs". Java is Unicode, right? Therefore it shouldn't depend on this option! The default value in the dialog is set during the Windows installation according to the Windows localization. For Russian-localized Windows that I've got it's "Russian (Russia)". It means that legacy A-versions of Windows API functions that take or return strings encode them in Windows-1251 encoding.

I intentionally didn't use anything that depends on file.encoding in the sample code. I also do not trust in System.out for non-ASCII output, in my experience it was always hit-or-miss depending on where the output is displayed. Not only JCL itself plays games with its encoding, IDEs and various shells also have conflicting opinions on the standard output encoding. So dumping arguments into a file with a known encoding isolates the problem to the main arguments values.

I also tested running the sample with "English (USA)" for "Language for non-Unicode programs" and got ?s both for classic and native runs. PowerShell was positively crazy in the process, I couldn't even type Russian in there.

from graal.

Ok, so I have discussed this internally: The JDK seems to convert arguments in their app launchers: https://github.com/openjdk/jdk/blob/700d2b91defd421a2818f53830c24f70d11ba4f6/src/jdk.jpackage/windows/native/common/WinSysInfo.cpp#L137

Instead of doing this, we can avoid the additional overhead (and potential for errors) by switching to wmain on Windows. This will also allow us to provide other features on Windows such as a javaw.exe like entry point that allows running an app without a command prompt.

We currently have no ETA for this but we will update this ticket when we do.

from graal.

Thanks a lot for bringing this to our attention, @ackasaber! Let me try to reproduce, debug, and fix this...

from graal.

Ok, so I'm having problems to reproduce this. I tried changing my system language but that did not seem to work. In a PowerShell, I can run this:

> python -c "import sys; print(sys.argv)" Привет, мир
['-c', 'Привет', 'мир']
> java "-Dfile.encoding=UTF8" LogArguments Привет, мир
> cat .\arguments.txt
??????
???

With and without -Dfile.encoding=UTF8, I only get ?s in the arguments file.

Do you have any idea what is going on?

from graal.

Ok, so I got this working after enabling the beta support for UTF-8, but also, the native executable seems to do the right thing (I added a simple System.out.println(List.of(args)); to your reproducer):

Could you please provide us with more details how to reproduce the inconsistent behavior you're observing?

from graal.

Your Python piece by the way works well independent of the "Language for non-Unicode programs", at least in cmd.exe. PowerShell has troubles displaying Cyrillic characters when this option doesn't match the argument language, but Python output is still okay somehow.

from graal.

Thanks for the info, @ackasaber. I can finally reproduce the issue after changing the system locale (there are just too many ways to changes languages on Windows these days)....now looking into a fix.

from graal.

Hi, there. I encounter a similar probleam today. Try many parameters and failed.
However, I manager to solve this probleam by "provide a parameter file which containing all non-asciicode" instead of directly provide all parameters via command line.

Thus, maybe we could just write a simple wrapper script to invoke the native image.

from graal.

@CJ-Chen A nice workaround! But not a good general solution for GraalVM.

I would be surprised if this bug gets a fix this year. It just can't be a priority: it's only in Windows while the majority of Java apps run on Linux + it's in command line parsing and the majority of Java apps don't do much of it.

from graal.

Apparently Microsoft does a U-turn with their encodings zoo and now promotes using UTF-8 for new applications using a dedicated manifest property. The introduced activeCodePage manifest property was introduced in Windows 10 Version 1903.

Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, then -A APIs typically operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes.

The articles goes so far as to call "Win32 API [that] might only understand WCHAR" legacy.

It's also possible to slap this manifest onto an existing exe. I'll try it in the meantime.

See also a blog post from Raymond Chen about this feature.

from graal.