AttributeError: 'FluentBundle' object has no attribute 'add_messages'

[plowman]

I'm shopping around for gettext replacements. Is the python fluent client (and fluent more broadly) still being maintained?

I mainly ask because even the happy path described in the docs does not work properly right now:

pip install fluent.runtime
python

from fluent.runtime import FluentBundle
bundle = FluentBundle(['en-US'])
bundle.add_messages("""""")
Traceback (most recent call last):
  File "/park/server/env/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-2a5756cb5134>", line 4, in <module>
    bundle.add_messages("""""")
AttributeError: 'FluentBundle' object has no attribute 'add_messages'

Looks like it was yanked out in the 0.3.0 "pre-release" and the docs weren't updated.

Anyway, Fluent seems cool, but I'd rather not go through the heartache of using it only to learn in 6 months that Mozilla has abandoned it. Thanks for any insight you have!

[spookylukey]

I'm also now very confused about this "pre-release" marker visible on the GitHub release page. These version have been released to PyPI as "0.2.0", "0.3.0" etc. so I don't understand how they can be described as pre-release, there is nothing in even the version number to indicate it is a pre-release. @Pike @stasm why have these been put on PyPI without the docs being updated?

[stasm]

Sorry about this. We're more and more invested in Fluent and we've been migrating more and more of Firefox to it (see https://arewefluentyet.com). Over the last few months we've focused primarily on the JavaScript and Rust implementations, however, and this has resulted in this unfortunate situation when python-fluent has been getting a bit less attention.

The good news is that we're planning to start using python-fluent to localize the mozilla.org website which is a Django app. I expect things to improve very soon.

@Pike is away this week. He's been more involved in python-fluent lately than I have, so I'd prefer to wait for him a few days, and make a plan to fix the current situation next week.

[plowman]

Thanks for the reply @stasm! That makes me much more confident that this will be actively maintained.

Until the docs can be updated, do you know by chance what is the expected way to invoke fluent right now?

I have this in python 3:

from fluent.runtime import FluentBundle, FluentResource

resource = """
-brand-name = TheApp

welcome-to-the-app = Welcome to { -brand-name }!
"""

bundle = FluentBundle(['en-US'])
bundle.add_resource(FluentResource(resource))
# 🤔🤔🤔
translated_string, warnings = bundle.format_pattern(bundle.get_message('welcome-to-the-app').value)
print(translated_string)

In addition to being a pretty awkward interface, that outputs two mysterious unicode characters:

The two square characters can't be pasted into the github comment box, but you can print the same thing by doing this:

print('Welcome to \u2068TheApp\u2069!')

Any thoughts? Thanks in advance.

[zbraniecki]

In addition to being a pretty awkward interface, that outputs two mysterious unicode characters:

Those are FSI/PDI - unicode directionality reset characters. They notify layout that the content of the inner part may be of different directionality than the outer text. For example an Arabic name in an English string etc.

You can read more about it here: https://github.com/projectfluent/fluent/wiki/BiDi-in-Fluent#syntax

[plowman]

Interesting @zbraniecki thanks for that explanation. It makes sense to me that this is necessary sometimes, if you're combining ltr/rtl strings.

Possibly dumb question though: don't we often know when the placeable text has the same directionality as the target language? In those cases do we need these characters?

My more practical concern is that I've only tested the string output in a couple places (bash, unit tests, console output in python), and in each case the handling of these characters is not great. It would be nice if I didn't have to worry whether every email/sms/push/etc. my app sends is going to show unicode ☐mysteryboxes☐ to end-users. Maybe there's no easy way to get around this though?

[zbraniecki]

don't we often know when the placeable text has the same directionality as the target language?

The assumption is that in most cases you're providing variables that come from an unknown source - either user provided (name) which may be of either directionality, or a known directionality to any locale (so, both to ltr and rtl).

In those cases do we need these characters?

If both source and variable locale is known, we could avoid that. There's an idea to do sth like:

from fluent.runtime import FluentBundle, FluentResource

resource = """
welcome-to-the-app = Welcome to { $foo }!
"""

bundle = FluentBundle(['en-US'])
bundle.add_resource(FluentResource(resource))
msg = bundle.get_message('welcome-to-the-app')
translated_string, warnings = bundle.format_pattern(msg.value, {
    'foo': FluentString('Jane', dir="ltr")
})
print(translated_string)

and in that case we could skip injecting directionality separators for ltr locales. But this hasn't materialize yet.

My more practical concern is that I've only tested the string output in a couple places (bash, unit tests, console output in python), and in each case the handling of these characters is not great.

Yeah, we tested it in the Web environment and modern layout engines usually support Unicode and in result support FSI/PDI.
Bash should support Unicode, but I haven't tested it myself.

For unit tests, two thoughts:

1. You should avoid testing the exact output. This will make your tests single-locale, and one day you'll want to test against different locales.
   In result, I'd recommend a test like:

translated_string, warnings = bundle.format_pattern(msg.value, { 'foo': 'Jane' })

assert_eq(translated_string.contains("Jane"), true)
assert_eq(len(warnings), 0)

over trying to test exact output.

2. If you really want to test exact string, you can set bundle = FluentBundle("en-US", use_isolating=False).

[zbraniecki]

Python 3 console seems to handle bidi chars for me quite well:

▶ python3
Python 3.8.2 (default, Feb 26 2020, 22:21:03) 
[GCC 9.2.1 20200130] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print('Welcome to \u2068TheApp\u2069!')
Welcome to ⁨TheApp⁩!

[stasm]

Until the docs can be updated, do you know by chance what is the expected way to invoke fluent right now?

Your example is correct, I'd just add an if message.value is not None guard to it:

from fluent.runtime import FluentBundle, FluentResource

resource = FluentResource("""
-brand-name = TheApp
welcome-to-the-app = Welcome to { -brand-name }!
""")

bundle = FluentBundle(['en-US'])
bundle.add_resource(resource)

message = bundle.get_message('welcome-to-the-app')
if message.value is not None:
    translated_string, warnings = bundle.format_pattern(message.value)
    print(translated_string)

Obviously, in the example above the guard is not really required, since we know that welcome-to-the-app has a value. In a real-world use, however, this assumption must be verified because messages may only have attributes and no value. You'd also want to check if the message have any attributes and handle them the way the app expects (e.g. to populate HTML attributes).

The API is a bit verbose on purpose. It's supposed to be a low-level API on top of which easier-to-use abstractions can be built which suit the needs of the app. Some discussion about it can be found on the Discourse and in the PR implementing the API in fluent.js.

[plowman]

Hey @zbraniecki @stasm thank you both for the thoughtful replies. Some quick responses:

If both source and variable locale is known, we could avoid that. There's an idea to do sth like:

...
translated_string, warnings = bundle.format_pattern(msg.value, {
    'foo': FluentString('Jane', dir="ltr")
})

I like that idea. It would be great if you could also specify the direction in the messages file so that you could declare your brand name once, translate it in some languages, and still have it get proper treatment everywhere in the app while still dodging the bidi markers where they're not doing anything because all the text is in the same direction.

This of course assumes still that we can't usually tell which direction the arg text is supposed to be in and do this automatically, which I'm still not fully convinced about, but also too ill-informed to argue properly.

It seems like if you have Arabic/Farsi/Hebrew/etc. characters you must know basically 100% of the time it should be written RTL. So there must be some other edge case that I don't understand where you can't tell the direction of a user-provided string and so have to always insert the BIDI markers, as opposed to only inserting them when you know the direction switches.

---

Yeah, we tested it in the Web environment and modern layout engines usually support Unicode and in result support FSI/PDI.

Got it, and the bidi markers are definitely invisible in a couple places for me as well. The specific areas I've had trouble with the bidi characters were inside the python console that's nested in PyCharm (where I get boxes) and in bash on ubuntu 18.04 (where I'm unable to fully delete a line which includes those characters). Both support unicode quite well in every case I've seen except this one.

This could be a unicode version issue or just an issue with various implementations of unicode, but in either case my main point is there is some tooling "cost" involved for anyone who wants to start using Fluent, where clearly lots of tools do not yet understand what to do with the bidi markers. If you can "postpone" that cost for people who are doing simple stuff until they start mixing language directions, you can probably help increase the number of people who will use Fluent.

---

The API is a bit verbose on purpose. It's supposed to be a low-level API on top of which easier-to-use abstractions can be built which suit the needs of the app.

Makes sense. And I'm definitely a fan of exposing a low-level API and keeping it consistent so people can do complicated things when needed. In this case, though, why not also expose a couple of higher level methods which do the simple thing?

Because gettext, which I dislike deeply, beats Fluent hands down when it comes to the API for the simplest possible case:

import gettext
en = gettext.translation('base', localedir='locales', languages=['en'])
en.install()
my_message = en.gettext('simple-message')

In Fluent you could just as easily do the something as simple (and much simpler in the plural and arg cases):

bundle = FluentBundle(['en-US'])
bundle.add_resource(resource)

my_message = bundle.translate('simple-message')
my_message_with_args = bundle.translate('simple-message-with-args', {'name': 'Jeffica'})

And all it would take is a few lines so that every user doesn't have to reinvent things:

# (Inside FluentBundle)
def translate(self, message_key, args=None, allow_warnings=False):
  message = bundle.get_message(message_key)
  if message.value is None:
    raise FluentException(f"Could not find message_key {message_key}")

  translated_string, warnings = bundle.format_pattern(message.value, args=args)
  if warnings and not allow_warnings:
    raise FluentException(f"Found warnings looking up key {message_key} with args {args}:\n" + warnings.join("\n"))

  return translated_string

As a bonus, you avoid hundreds of duplicate questions from people like me showing up and asking the same question about how to actually implement Fluent. 😉

[stasm]

In this case, though, why not also expose a couple of higher level methods which do the simple thing?

This is a good idea. I think it would be best to expose these higher level methods on something higher up than FluentBundle. I like the idea of FluentBundle's API surface being very small, unopinionated, building-blocks-like.

We've been thinking of standardizing the high-level abstraction (called Localization) which would operate on an ordered sequence of FluentBundles (corresponding to the order of user's preferred languages), exposing easy-to-use methods like format_value and format_attribute. You can see the examples of such abstraction in fluent-dom (JS), in fluent-fallback (Rust), or in Luke's django-ftl. So far we taken an exploratory approach: each implementation has its own Localization. We're planning to spend some time this year learning from all these different approaches and unifying them. I hope this effort will start in a few months. I haven't written anything down on the subject yet; I'll probably start a thread on our Discourse at some point about this.

[Pike]

There are a couple of things here.

For one, @spookylukey is the only one on rtd that has access, and is involved here. The Pike over there isn't me. (I'm DrAxelHecht on rtd.) Luke, could you add me and remove that other Pike? Thanks.

The other is that we haven't merged the next-gen-api branch to master yet. Now that Luke gave his OK in https://discourse.mozilla.org/t/experimental-release-of-python-fluent-runtime/47569, we should get that done quickly.

The new docs themselves are written, you can get a sneak peek on https://github.com/projectfluent/python-fluent/blob/next-gen-api/fluent.runtime/docs/usage.rst. Note that that highlights the new API around the Localization class, and (hopefully) clarifies that Bundle is only used to implement Localization-type classes.

Happy to follow up on the other topics raised here on discourse.

[spookylukey]

For one, @spookylukey is the only one on rtd that has access, and is involved here. The Pike over there isn't me. (I'm DrAxelHecht on rtd.) Luke, could you add me and remove that other Pike?

Done!

[Pike]

OK, I've updated the configuration here for the webhook to work again, and rebuilt the docs based on master. The next-gen-api branch is merged, too, so the docs now reflect what's on pypi.

As I mentioned, the other comments are better suited for discourse, so I'm closing this out.