Document UTF-8 Conversion in Spec + Simplify marshall/unmarshall Implementations. about colfer HOT 9 CLOSED

I think text conversion depends on the implementation, i.e., the rules are not related to the data format. The compiler manual (see README) states the following.

	Text validation is not part of the marshalling and unmarshalling
	process. C and Go just pass any malformed UTF-8 characters. Java
	and JavaScript replace unmappable content with the '?' character
	(ASCII 63).

So java.lang.String is no longer backed by a char(acter) array. With the new implementation it is even harder to access the data in an efficient way. 😖 Happy to hear about better alternatives for String#charAt(int). String#getBytes allocates memory. The unmarshaller uses String(byte[],int,int,java.nio.charset.Charset) now, and that works fine.

No external libraries for generated code is key!

Feel free to open an issue for a specific improvement idea.

from colfer.

Had a quick look at the new streams with String#chars. It is way slower 😱than String#charAt(int).

from colfer.

1. The size fit is the maximum ratio from UTF-16 char(acters) to UTF-8 bytes. That is, encoding of a char costs 1, 2 or 3 bytes; never more.

2. The golden cases have a string with all cases covered.

from colfer.

What do you mean with "for now"? 😬 This must hold forever, even with malformed UTF-16 sequences.

from colfer.

--- a/ecma/test.js
+++ b/ecma/test.js
@@ -50,6 +50,7 @@ function newGoldenCases() {
                '87ffffffffffffffff2e5da4e77f': {t: new Date(-223), t_ns: 888999},
                '0801417f': {s: 'A'},
                '080261007f': {s: 'a\x00'},
+               '0804f0908d887f': {s: '𐍈'},
                '0809c280e0a080f09080807f': {s: '\u0080\u0800\u{10000}'},
                '08800120202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020207f': {s: '                                                                                                                                '},
                '0901ff7f': {a: new Uint8Array([0xFF])},

… passes just fine.

from colfer.

Thank you for clarifying. So here are some specific suggestions:

1. Java's size fit should probably use *4 or *5 as a general rule. Assuming 𐍈🐱‍🐉🤷‍♂️🥗🚂 emojis are the norm than the exception.
2. We should probably add ser/deser tests with UTF characters across all 4 character ranges in these tests.

from colfer.

Ah, yes, based on characters, not code points. That should be okay for now.

from colfer.

For now, it meant, until, Unicode ups the range dramatically.

from colfer.

Unicode doesn't up the range dramatically. It would would also be against their own stability policy.

from colfer.