projectfluent / fluent.js Goto Github PK

Sync XHR is deprecated. We shouldn't use it to download localization resources. If FOUC is a concern, let's research other avenues for improving the perf. For instance:

• the display: none trick,
• rel=prefetch and rel=preload
• server-side language-negotiation
• build-time optimizations for above-the-fold content
• others?

The main motivation behind formatToParts was the use-case of passing React elements into translations in fluent-react. With a plan to implement overlays in fluent-react (#103) this use-case goes away.

formatToParts has been buggy and under-spec'ed. Let's remove it for good. It will also make the interface of MessageContext very simple: the format method always returns a null or a string.

Any reason multiline comments are not apparently supported?

Coming back from the Unicode Conference, there was a lot of chatter about pseudo-locales.

Fluent already had a pretty good support for pseudo-locales in the past and due to our client-side mode, we offer an exciting approach to pseudo-locales - runtime pseudolocalization.

I'd like to bring back this: https://github.com/l20n/l20n.js/blob/v3.x/src/lib/pseudo.js to modern fluent.

@stasm - do you have any thoughts on how would you like it to work?

Currently we have Message References.

brandName = Loki
installing = Installing { brandName }.

menu-save = Save
help-menu-save = Click "{ menu-save }" to save the file.

This indicate the translator to not translate the what inside the { }.

I suggest we can add another syntax to make the contents to be translated too.
For example

alert_message = Click #{ here } for more detail.

This is useful because we can keep the sentence to be translated all together. And if we provide AST form of the translated result, we provide the renderer ability to further enhance the appearance or behaviour of the word here.

This should be possible:

key1 = {{ foo }}
key2 = {     { foo }     }
key3 = 
    {
        { foo }
    }

I have the following (simplified, I know the link should also be localized, but first things first) code:

<Localized
  id='door-code-explanation'
  $contactLink={<a href='mailto:[email protected]'>Contact us</a>}>
  <p>
    This code will be surfaced to people attending the upcoming session.
  </p>
</Localized>

// ftl
door-code-explanation = 
    This code will be surfaced to people attending the upcoming session.
    { $contactLink } if you have further questions.

what I would expect to see:

This code will be surfaced to people attending the upcoming session. Contact us if you have further questions.

instead I see

This code will be surfaced to people attending the upcoming session. contactLink if you have further questions.

There are no errors in the console.
Also, when I change the $contactLink prop to a regular string, the result is as expected.

This should not be valid:

key = Value
    #foo bar

I've started working on a Rust port of fluent-langneg and started thinking more about the role of this package in the Fluent ecosystem.

While all it does on the surface at the moment is negotiate languages, the package provides a more complete functionality including:

• Opinionated definition of locale id, locale range, extension keys etc.
• Ability to parse and serialize the locale id
• Ability to handle language tags
• Opinionated logic of negotiating locales and languages

This functionality is necessary for Fluent to work, and it fits into CLDR/ICU logic, but it goes beyond just negotiation.

For example, in ECMA402 we're currently working on Intl.Locale API which will be very similar to what we did in fluent-langneg for the Locale class (maybe because I'm the author of the proposal, who knows).

I believe that even raw functionality like parsing/serializing and operating on locale tags is useful beyond just negotiation. In the current fluent-langneg it can be easily achieved by exposing the Locale class, and soon we'll be able to possibly just replace it with calls to Intl.Locale (I'm not 100% sure if we will, because Intl.Locale may not support ranges in the first version).

For fluent-langneg-rs it would make sense to expose more than just language negotiation, so I believe that fluent-locale-rs is a better name for the package. Same for the python version.

I know it's a hassle to change the name, but I think it makes sense to do it in this case.

This causes fluent-reeact to not work on ie 11 :) It's probably ok if this is a WONTFIX, but I'm pretty sure Babel can be configured to provide a Symbol polyfill, so it might actually be easy to fix.

I spent a couple of hours today trying to figure out what was wrong with my code. I finally discovered that the issue is that the parser for FTL formatted strings does not gracefully handle indentation. For example:

const types = new MessageContext('en-US');
types.addMessages(`
  sets = {$count ->
    [one]   set
   *[other] sets
  }
  repeats = {$count ->
    [one]   repeats
   *[other] repeats
  }
`);

Throws the error Expected an entry to start at the beginning of the file or on a new line. It took me quite some time to figure out the error was in the whitespace, not in the newlines.

On the other hand, this works at the cost of poorer readability:

const types = new MessageContext('en-US');
types.addMessages(`
sets = {$count ->
[one]   set
*[other] sets
}
repeats = {$count ->
[one]   repeats
*[other] repeats
}
`);

I understand that this may or may not follow the FTL specs, but I think any developer will agree that improved readability through indentation is a good thing.

The react-native-l20n package by @jamesreggio works perfectly in this regard if anyone would like to see an example of how it is implemented.

I'm curious to hear everyone's thoughts, thanks.

Hi,

I'd be curious to know how this project compares to the Intl.MessageFormat proposal/library and whether it may be compatible (or intended to become compatible) and how the features of each compare, etc.?

Thanks!

It would help me to work with the parser tests to know how much of the parser is covered with tests. Istanbul seems like a good candidate? https://github.com/gotwarlost/istanbul

Currently, JSDoc generates output which isn't very useful. The classes are not extracted and all methods end up in the global scope. I think it's related to jsdoc/jsdoc#1132. The fix might be to use the @module tag consistently.

In cc32a1e we started catching errors thrown by Intl formatters inside of FluentType. Should we report them somehow?

In

@zbraniecki noticed something weird about this code:

I looked into it. The code is broken and possibly never worked in the first place. The only reason why it doesn't go into an infinite loop is that a few lines before there's this:

translationElement.removeChild(childElement);

Which means that by the time we get into getIndexOfType, element is always the first sibling.

Babel 7 is still in beta but very much usable. Let's use this issue to track what changes are needed on our side when we upgrade.

Improvements:

• Remove the transform-builtin-extend plugin. (babel/babel#7020)
• OK to use destructuring in for await loops. (babel/babel#5932)
• Maybe drop stage-3 in favor of @babel/plugin-proposal-async-generator-functions?

Open questions:

• Will the logic of reading .babelrc files change? (babel/babel#6766)

Following instructions to set up and test the fluent packages results in tests failing

also, this line is preventing some tests from running: https://github.com/projectfluent/fluent.js/blob/master/fluent/test/functions_builtin_test.js#L86

with that .only included, one test fails, by removing it, 6 tests fail.

I noticed two issues related to placeables at the beginning and the end of line:

1. In a multiline message, a placeable at the beginning of the line results in an error, while it should be allowed.

2. In a multiline message, a placeable at the end of the line consumes the \n after it.

I created fixtures for tests: stasm@f195581

(The branch is at https://github.com/stasm/fluent.js/tree/fixtures-placeable-line-extremes.)

In cc32a1e we started catching errors thrown by Fluent functions (like built-ins) in Call Expressions. Should we report them somehow?

Once we get a good corpus of tests for the parser, I'd like to try to replace the current runtime parser with a stream parser. :till says that it would be a perfect use case, and I can see a benefit of doing that.

This will not be ready for production until streams are widely available, but may be useful for our perf. critical implementations

https://github.com/whatwg/streams

As of fluent-react 0.4.1 it is possible to pass React elements as props to <Localized> to interpolate them into the translation:

<Localized
    id="sign-in-or-cancel"
    $signInButton={
        <Localized id="sign-in-button">
            <button className="action" onClick={signIn}>{'Sign in'}</button>
        </Localized>
    }
    $cancelButton={
        <Localized id="cancel-button">
            <button className="text" onClick={cancel}>{'cancel'}</button>
        </Localized>
    }
>
    <p>{'{ $signInButton } or { $cancelButton}.'}</p>
</Localized>

sign-in-button = Sign in
cancel-button = cancel
sign-in-or-cancel = { $signInButton } or { $cancelButton }.

This is a bad localization practice because it results in the translation being split into multiple strings and then interpolated.

Instead we should do something similar to fluent-dom's overlay logic.

Solution

With overlays, the translation can include some HTML which will be parsed by Fluent. HTML children found in the translation are then matched against child elements in the source. In React, we can pass the source elements via props.

<Localized
    id="sign-in-or-cancel"
    button={
        <button className="action" onClick={signIn}></button>
    }
    a={
        <button className="text" onClick={cancel}></button>
    }
>
    <p>{'<button>Sign in</button> or <a>cancel</a>.'}</p>
</Localized>

sign-in-or-cancel = <button>Sign in</button> or <a>cancel</a>.

In 0.0.2, require("fluent-langneg") exported the single function negotiateLanguages.

In 0.0.3, the export is an object of the form { default, acceptedLanguages }, where negotiateLanguages is assigned to default.

This breaks existing clients that loaded the main export node-style:

const { negotiateLanguages } = require('fluent-langneg')

Same issue occurs with the compat builds. Not a huge deal, but would have been nice to see a larger version number bump to signify API changes. Would be great to see the new API added to the README, too 👍

Howdy,

Just a very few quick questions based on a quick look at the repo (which I think would be more inviting for others digging in and getting their hands dirty with the code)...

1. Might you consider a cross-platform, wholly JavaScript-based build routine rather than Make files?
2. Why no public npm repo? (the 0.0.1 version ought to be enough to flag its alpha state)
3. How about a module field in package.json to indicate the Rollup entry file as well as main for any Node.js-based entry file usage?

Thanks!

It would be great to allow async iterables returned by generateMessages. We'd need to move l10n.getMessageContext out of Localized.render and possibly cache it?

I've written and rewritten this issue more times than I'd like to admit, all because I don't want to sound ungrateful for this otherwise amazing project. I'll try to keep this short, as none of us really wants to support IE, but some of us don't have a choice...

This issue is related to #79 but it's more of a "feature" request, rather than a bug report.

IE11 requires a significant amount of polyfills to run even the compatibility bundle, thanks to primarily for...of, which in order to be transpiled safely, depend on the use of Symbol.iterator, which requires a babel-polyfill (50kb+ gzipped).

By not using for...of, you will not only decrease the size of your transpiled bundle, but also significantly reducing the number of polyfills IE11 requires to load. In most cases, for...of only saves 2/3 lines of code compared to other loop constructs, but those 2 lines are dramatically offset by the size of the transpiled code and polyfills.

I know this is a big thing to ask, but is not using for...of something you would consider for this project?

Thanks for reading this far and thank everyone on the team for this amazing project.

I'm filing this in case we decide against requiring = after the identifier (projectfluent/fluent#63, projectfluent/fluent#73). We'll still need to fix the current (0.4.2) parsing behavior which parses:

key =
    .attr = Attribute

as a message with the value of .attr = Attribute (both parsers).

In 0.4.1 the above code was parsed as a null value; 7e52517 changed it.

Given our EBNF it might make sense to rewrite the full AST parser as an LL(1) parser combinator.

I found the following two resources very helpful in understanding the principle:

http://www.garshol.priv.no/download/text/bnf.html
https://medium.com/@chetcorcos/introduction-to-parsers-644d1b5d7f3d

fluent version 0.4.1

// ftl file
today-is = today is { DATETIME($date, month: "long", day: "numeric") }

// React component
...
  <Localized id='today-is' $date={new Date()}>
    <p>today is {(new Intl.DateTimeFormat("fr", {month: "long", day: "numeric"})).format()}</p>
  </Localized>
  <p>today is {(new Intl.DateTimeFormat("fr", {month: "long", day: "numeric"})).format()}</p>
...

result

node version 6.11.2

When the Accept-Language header value is '*', negotiateLanguages seems to return the first three locales, plus the default:

 negotiateLanguages('*', 
  ['ach', 'ar', 'be', 'ca', 'cs', 'da', 'de', 'dsb', 'el', 'en-US', 'cy', 'eo', 'es-AR'],
  { defaultLocale: 'en-US'})
// returns ['ach', 'ar', 'be', 'en-US']

Related to mozilla-services/screenshots#3231

https://bugzilla.mozilla.org/show_bug.cgi?id=1410857

Right now, the formatting methods of Localization iterate over the the list of MessageContexts first and the list of requested ids second. In other words, for each context we try to retrieve a full list of translations. Once we have that list, we need logic to discover if there have been any errors and to decide if fallback is required.

This approach has a few limitations:

• it's hard to implement the fallback to the id in case a message is missing from all contexts (bug 1410849),
• ideally, the fallback to the id would only happen if the bindings (e.g. fluent-dom) allow it; for some known lists of DOM elements the bindings might prefer to keep the value empty and fall back in one of the attributes.
• it forces a two-pass iteration in fluent-dom: first we iterate to get translations for all ids across all known contexts, then we iterate over all DOM elements to apply the translations.

I'd like to propose an alternative: let's switch the order of iteration to first iterate over all ids (all DOM elements in fluent-dom) and the for each id, use mapContext to get the best context for this id.

fluentfmt.js incorrectly absorbs keys into preceding comments under certain circumstances. I'm not quite sure of the exact parsing bug, but here's a test case to experiment with:

test.ftl:

// Foo page

fooPageTitle = Foo

// Test page

testPageHelloWorld = Hello, world!

Output of fluentfmt.js test.ftl:

// Foo page

fooPageTitle = Foo
// Test pagetestPageHelloWorld = Hello, world!

Right now, fluent-react uses

<LocalizedElement id="hello-world">

Now, jsx looks like XML, and the id attribute has connotations about being unique in the source document, whereas this id is only unique in the MessageContext.

@stasm and I chatted about that, and we could use message instead:

<LocalizedElement message="hello-world">

In fluent-dom it is currently not possible to reorder child elements found in the translation if they're of the same type. An example from bug 1424682:

# This string is shown to notify the user how to enable an extension that they disabled.
extension-controlled-enable = To enable the extension go to <image/> Add-ons in the <image/> menu.

Pseudolocalizations were useful for debugging in Firefox OS. I'd like to research how to implement them in Fluent. It's trickier than it looks: if we transform deep inside Fluent, we'll break HTML content in translations. If we transform late in bindings, we'll likely transform interpolated values, too.

My current idea is to allow passing a transform function into the MessageContext constructor:

const cx = new MessageContext('en-US', { transform: pseudolocalizeFn });

If present, this function would be applied to all TextElements.

Bindings would be responsible for detecting that the current locale code is in fact a pseudo-locale (e.g. en-x-psaccent) and for passing the proper real locale code into the MessageContext constructor together with the transform function.

Hi I am coming from here:
projectfluent/fluent#78

the solution that @stasm pointed there woked fine only for strings:

// Destructure the array of positional arguments to flag_value.
FLAG([flag_value]) {
    return flag_value? 'active' : 'inactive';
}

flag_value becomes something else (FluentNone) when it is false and becomes something else (FluentNumber) when it is for example 0

Looking to:

I could use value() method to get the value... but that does half of the trick because FluentNone value is the same name/id of the variable.... :'(

// Destructure the array of positional arguments to flag_value.
FLAG([flag_value]) {
    return value(flag_value)? 'active' : 'inactive';
}

function value(arg) {
  // StringExpression-typed options are parsed as regular strings by the
  // runtime parser and are not converted to a FluentType by the resolver.
  // They don't have the "value" property; they are the value.
  return typeof arg === 'string' ? arg : arg.value;
}

For some reason you are passing those objects to the function... I am completely out of context here... but just looking it from outside. Couldn't it be just enough for these functions to receive as input just the real value.... so it wouldn't be needed to know all those details when implementing these simple functions?

Based on the work done in https://github.com/projectfluent/fluent/milestone/4 and my first POC patches I believe we should land the changes in the following order:

• tags removal (#110)
• comments switch + section removal (#109)
• private messages (#130)
• enforced = after id (#132)

Would be nice if this were treated as in markdown that you can make a multiline string and the whitespace is only converted to a newline explicitly, for example by adding a blank line,
so for example

message = Loki is a simple micro-blogging
    app written entirely in <i>HTML5</i>.
    It uses FTL to implement localization.

currently results in "value": "Loki is a simple micro-blogging\napp written entirely in <i>HTML5</i>.\nIt uses FTL to implement localization."

But might be better if it resulted in "value": "Loki is a simple micro-blogging app written entirely in <i>HTML5</i>. It uses FTL to implement localization."

and perhaps

message = Loki is a simple micro-blogging
    app written entirely in <i>HTML5</i>.

    It uses FTL to implement localization.

could result in "value": "Loki is a simple micro-blogging app written entirely in <i>HTML5</i>.\nIt uses FTL to implement localization."

Just a thought, would be happy to hear other ideas, but currently I can imagine developers wanting to keep line-width down for file aesthetics, and not necessarily realizing they are introducing newlines.

The spec is quite lenient wrt. to allowing newlines: https://github.com/projectfluent/fluent/blob/master/spec/fluent.ebnf

Also see: projectfluent/python-fluent#15

When translating complex pages on a website, we ran into a problem where we have variable content that contains HTML markup, that is passed as l10-n-args attribute values in the DOM. This content is fro a trusted/vetted source, so we don't need the sanitizeArgs escaping. However, there doesn't seem to be a way to avoid it. Since the args are sanitized when the keys are fetched from context, before they are passed to translation, I don't see a way to add a special TrustedText type. I was considering a builtin function HTML() or some such that would reverse the santiizeArgs replacements.

Hi there!

I got this error on safari 9.1

It looks like you use this line on the parser.js
const identifierRe = new RegExp('[a-zA-Z_][a-zA-Z0-9_-]*', 'y');

the sticky flag is not supported...
I have seen this but I haven t tried yet
https://www.npmjs.com/package/babel-plugin-transform-es2015-sticky-regex

Since fluent-langneg runs on both server and client, it would be nice to not have to separately parse the Accept-Language header string into an array of locales:

negotiateLanguages(req.header('Accept-Language'), availableLocales)

As things stand, I've either got to add another language negotiation library, like accept-language-parser, or roll my own, probably buggy, regex to do the job ;-)

In projectfluent/fluent#51 the syntax was changed to only forbid ? at the start of Identifiers.

str:

clone fluent, go to fluent-react examples folder, npm i and npm start

the resulting page is blank with the following console error:

From my review of the tutorial and API docs:

1. It looks like http://projectfluent.org/fluent.js/fluent/ may need to be updated as it's referring to http://projectfluent.io/ which is giving a 404
2. I see on MDN one argument missing from http://projectfluent.org/fluent/guide/functions.html#datetime : hourCycle
3. Under the tutorial docs for Number, I am not clear on the meaning of this one sentence: "The second example may be used to pass additional formatting options to the NUMBER formatter for the purpose of choosing the correct plural category".
4. For https://github.com/projectfluent/fluent/wiki/Fluent-and-ICU-MessageFormat , I think you want the examples with Fluent.NumberArgument to instead use Intl.MessageNumberArgument
5. Coming through https://github.com/projectfluent/fluent.js/tree/master/fluent-dom , one doesn't get automated documentation of DOMLocalization or Localization such as is within https://github.com/l20n/l20n.js/blob/master/docs/dom_localization.md and https://github.com/l20n/l20n.js/blob/master/docs/localization.md -- which would be nice to have available from fluent-dom.
6. I think it could help to have some clarification about the differences between the seemingly overlapping projects (and overlapping APIs), fluent-web (which isn't referenced on the main README), fluent-dom, and the separate l20n.js project (which incidentally sees like it should be dependent on fluent-dom but isn't).
7. I think the above-referenced API documentation could also probably benefit from documenting the available built-in classes (MessageArgument, MessageNumberArgument, and MessageDateTimeArgument) for use as part of partial arguments within format (if not the other exports, mapContextSync and CachedIterable) since looking through the API docs alone didn't make this apparent. If l20n.js' use of List (apparently Intl.ListFormat) as referenced at http://l20n.org/learn/builtins is also available, it would be nice to have this documented.
8. If l20n.js' use of | for message line breaks is also available in fluent.js, it is not documented as such at http://projectfluent.org/fluent/guide/text.html (as it is at http://l20n.org/learn/working-with-text-multiline-interpolation ). (And if it isn't, I think it should be drawn out as a special feature of l10n)
9. If l20n.js' use of a built-in LEN as documented http://l20n.org/learn/advanced-selectors is also available in fluent.js, it is not documented at http://projectfluent.org/fluent/guide/functions.html#built-in-functions (and if it isn't, I think it should be drawn out as a special feature of l10n). Also TAKE is mentioned in the example code at http://l20n.org/learn/complex-example without detailed explanation there, and it is not clear whether it is available by default through l20n.js or through fluent.js. (Moreover, http://l20n.org/learn/complex-example is surfacing a TypeError on the page but as there is no issue tracker for https://github.com/l20n/l20n.js I can't report it there.)
10. In the example at https://github.com/projectfluent/fluent/wiki/Get-Started#high-level-api, the second new Localization should instead be new DOMLocalization and the first is missing the required second argument.
11. I'd think another use of Terms that might be called out is use of aliases for representing repeated Unicode (as XML entities did). For example, I'd presume one could define -ellipsis, -copyright, etc. for these symbols if repeating their use, with the use of Terms here designating they were just local definitions perhaps set up by the translator for their own convenience.

After #38 landed, I'm seeing the following warning produced by rollup when building fluent-langneg:

⚠️ Using named and default exports together. Consumers of your bundle will have to use FluentLangNeg['default'] to access the default export, which may not be what you want. Use exports: 'named' to disable this warning
https://github.com/rollup/rollup/wiki/JavaScript-API#exports

We can either follow the advice in the warning and add exports: 'named' to rollup's config, or consider making nagotiateLanguages a named export.

Currently the following forms are possible:

import negotiateLanguages from 'fluent-langneg';
import negotiateLanguages, { acceptedLanguages } from 'fluent-langneg';

If nagotiateLanguages becomes a named export:

import { negotiateLanguages, acceptedLanguages } from 'fluent-langneg';

@zbraniecki Do you expect fluent-langneg to grow wrt. its API? For instance, will we want to add something to the effect of filterSupportedLanguages to cater to bug 1358628?

It shouldn't be possible for localizers to break React by defining content of void elements:

foo = <input>This is invalid content</input>

The W3C defines the following list of void elements:

area, base, br, col, embed, hr, img, input, link, meta, param, source, track, wbr

There's also a package which maintains an up-to-date list of void elements: https://github.com/pugjs/void-elements/blob/master/index.js

version 0.4.1 - using latest Firefox and Chrome browsers.

If I pull all components from fluent-react, I get console warnings:

Warning: Failed context type: The l10n context field must be an instance of ReactLocalization.

the warning provides a stack trace to a component using fluent-react components, if I change the import for fluent-react in that component to pull from fluent-react/compat, the error goes away.

It's not entirely clear what needs to happen to allow for imports from the base library.

Both fluent-web and fluent-dom are at versions 0.0.1. I expect the API to fluctuate before it stabilizes. Let's use relative imports for the while being so that it's easier to iterate.

Once we reach 0.1, we can bring back package imports and proper dependency management via package.json.