`jbang export gradle` and UTF-8

[wfouche]

When experting a UTF-8 Java applications to a Gradle project, Unicode characters are not displayed correctly when though Windows is using code page 65001.

Step 1 - run Java program with JBang

C:\> jbang run JvmRuntimeOpts.java
--- 🚀 JVM Runtime Options (VM Arguments) ---
No explicit JVM Runtime Options found (default settings are in use).

--- 📝 Application Arguments ---
No Application Arguments were passed to the main method.

--- ✅ Execution Complete ---

Step 2 - export to Gradle run using Gradle

C:\> jbang export gradle JvmRuntimeOpts.java
C:\> cd  JvmRunTimeOpts
C:\> .\gradlew run
--- ? JVM Runtime Options (VM Arguments) ---
Option 1: -Dfile.encoding=UTF-8
Option 2: -Duser.country=ZA
Option 3: -Duser.language=en
Option 4: -Duser.variant

--- ? Application Arguments ---
No Application Arguments were passed to the main method.

--- ? Execution Complete ---

The fix is to also add Java runtime options:

• -Dstdout.encoding=UTF-8
• -Dstderr.encoding=UTF-8

[wfouche]

With the PR applied the output from Gradle is:

--- 🚀 JVM Runtime Options (VM Arguments) ---
Option 1: -Dstderr.encoding=UTF-8
Option 2: -Dstdout.encoding=UTF-8
Option 3: -Dfile.encoding=UTF-8
Option 4: -Duser.country=ZA
Option 5: -Duser.language=en
Option 6: -Duser.variant

--- 📝 Application Arguments ---
No Application Arguments were passed to the main method.

--- ✅ Execution Complete ---

[maxandersen]

Is this true for recent java and Gradle versions ?

[wfouche]

Yes, using Java 21.0.9 and Gradle 8.14.3

[wfouche]

Java 19 introduced -Dstdout.encoding=UTF-8 to set System.out to UTF-8.

https://www.oracle.com/java/technologies/javase/19-relnote-issues.html

core-libs/java.lang
➜ New system properties for System.out and System.err (JDK-8283620)

Two new system properties, stdout.encoding and stderr.encoding, have been added. The value of these system properties is the encoding used by the standard output and standard error streams (System.out and System.err).

[wfouche]

Maybe Jbang needs a --utf8 command-line option ....

[maxandersen]

great, so not even something thats consistent across java versions :)

Also just bumped into https://github.com/orgs/jbangdev/discussions/1399 which is about encoding of the source files.

This is just a Windows vs Java induced mess :) not sure what the right/best solution is. --utf8 would at least allow a bit of logic to ensure consistent intended behavior instead of adding flags that aren't relevant in all OS/Java combos.

[wfouche]

--utf8 would at least allow a bit of logic to ensure consistent intended behavior instead of adding flags that aren't relevant in all OS/Java combos.

Yip.

A workaround is to set JDK_JAVA_OPTIONS before running Gradle, but then an additional message is printed to the console when Gradle is run:

• NOTE: Picked up JDK_JAVA_OPTIONS: -Dfile.encoding=UTF-8 -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8

C:\>  set "JDK_JAVA_OPTIONS=-Dfile.encoding=UTF-8 -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8"

Being able to add a //UTF8 directive to JBang scripts would be nice.

[quintesse]

I haven't even really looked at this but it just sounds completely wrong. In what world is that option not always set to "true"?
And if there's a reason and we can't figure it out by ourselves but the user has to somehow determine when to use it ... we've already lost I'd say.

I'd say that adding flags to OS/java combos where it's unnecessary but doesn't do any harm would be much preferable over magic options that uses should try to see if it fixes their problem or not. That sounds like "try rebooting" to fix computer problems.

[wfouche]

JBang cannot assume that it should exclusively use UTF-8 for console out. Someone might write a UTF-16 application in Java with the expectation of being able to run it with JBang.

import java.io.PrintStream;
import java.io.UnsupportedEncodingException;
import java.nio.charset.StandardCharsets;

public class Utf16ConsoleDemo {
    public static void main(String[] args) {
        try {
            // Reassign System.out to a new PrintStream using UTF-16 encoding
            System.setOut(new PrintStream(System.out, true, StandardCharsets.UTF_16.name()));
            
            String unicodeMessage = "Hello, world! こんにちは世界 ✨✨✨";

            // Print the string - Java handles the internal UTF-16 to output UTF-16 conversion
            System.out.println(unicodeMessage);

        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
    }
}

//UTF8 would just setup System.stdout and System.stderr to be UTF_8 steams, and also set other UTF-8 options and could also set the console to UTF-8 mode as well (on Windows).

So I still think //UTF8 would be a useful option to have in JBang.

And I agree that it would (should) nearly always be specified.

[maxandersen]

the issue is that these encoding settings applied on all platforms aren't without issues.

If we lets say applied all -Dencoding flags we would need a way to turn it off..like a //NO-UTF8 :)

...but maybe a //ENCODING= which we then take and apply everywhere...and have a //UTF8 shorthand that is equivalent to //ENCODING=UTF8 ....but users can also do //ENCODING=UTF-16 ...or something different.

and if you need different encodings for files vs streams, etc. and just rely on OS defaults you dont put anything?

...could also do a //!ENCODING ..and assumed //ENCODING=UTF-8 for all...but that would mean java/javac gets called with lots of flags that might not be what you need....but I must say I'm inclined to do it that way (enforce utf-8 unless told otherwise)

[quintesse]

But does setting the encoding to UTF8 even prevent you from writing code that uses UTF16? Aren't you simply setting Java's default?