Bug #13181

Languages with 3-letter codes cannot be added in AtoM's user interface settings

Added by Dan Gillean over 1 year ago. Updated 4 months ago.

Status:In progressStart date:09/24/2019
Priority:MediumDue date:
Assignee:Dan Gillean% Done:

0%

Category:I18NEstimated time:16.00 hours
Target version:-
Google Code Legacy ID: Tested version:2.5, 2.6
Sponsored:No Requires documentation:

Description

If you look at the list of supported languages in AtoM and their corresponding language codes, you'll see that almost half of them have 3-letter language codes. Examples include:

  • ace - Achinese
  • ach - Acoli
  • ada - Adangme
  • ady - Adyghe

...and so on.

When attempting to add these in Admin > Settings > i18n languages, the page will refresh after submitting, but no new language is added, depsite the fact that these are coming from symfony defaults.

To reproduce

  • Log into AtoM and navigate to Admin > Settings > i18n langauges
  • Pick one of the languages with a three-letter language code, such as Achinese, in the languages selection drop-down
  • Click the Add button

Resulting error

  • The page reloads, but no new language is added

Expected result

  • New language is added to the user interface

Related issues

Related to Access to Memory (AtoM) - Bug #13222: Build SQL task changes the size of culture columns Verified 12/05/2019
Related to Access to Memory (AtoM) - Bug #13463: Problem: CSV import with pipe separated culture data cras... New 01/21/2021
Related to Access to Memory (AtoM) - Bug #13465: Problem: users can add blank language rows in the i18n se... Verified 01/26/2021

History

#1 Updated by Mike Cantelon over 1 year ago

The format_language call on line 45 of apps/qubit/modules/settings/actions/languageAction.class.php seems to be throwing an error. In that function the sfCultureInfo::getInstance seems to error for these languages.

#2 Updated by José Raddaoui Marín over 1 year ago

  • Status changed from New to In progress
  • Assignee set to Dan Gillean

Hi Dan,

To follow up on Mike's comment, the problem occurs when the new language code is validated/set in here. The regex used to validate the culture looks for a 2 chars language code (and optionally the country code) and it doesn't allow a 3 chars language code. From the Symfony docs, that's the way the culture should be formatted ...

The language is coded in two lowercase characters, according to the ISO 639-1 standard (for instance, en for English). The country is coded in two uppercase characters, according to the ISO 3166-1 standard (for instance, GB for Great Britain).

I think the main problem comes from mixing languages with cultures. The i18n languages form uses the languages available in the culture data, which includes those 3 chars codes and the language name in the current culture, but those languages are not a representation of the cultures available in Symfony. My first impression is that those lists of languages are just in there for localization (L10n) and not for internationalization (i18n); and that we should be using a list of Symfony cultures in the settings form.

This is the current list of languages showed in the form and this would be the list of available cultures in Symfony. However, I think some of those cultures won't match the regex mentioned above and we probably don't want to show them all, so this list may need some filtering.

To be fair, the issue is caused because the language code is used as the culture code to format the language, which could be easily fixed and would allow the i18n languages form to work properly. But then, the application will try to use the language code as the sf_culture and that won't find the related translation files as those are named by the culture code (see the main translations folder).

Please, let me know your thoughts and I'll look for an estimate.

#3 Updated by Dan Gillean over 1 year ago

Hi Radda,

I didn't fully understand the last part, but: it sounds like if we follow your first suggestion, we will essentially lose about 200 languages in the settings? I.... guess we don't have to worry about people using (or translating for) these languages, since they can't currently be loaded in AtoM anyway? Or is there a risk that we will affect existing users?

Also, if we do go with the first solution, and start using the right culture list, would it be possible for you to make something like Sevein did previously, here?

That is, a list we can share with the community that includes both the symfony language name and the corresponding culture code?

I would like to hear from other AtoM teammates before we proceed since it sounds like this could impact existing users...

#4 Updated by José Raddaoui Marín over 1 year ago

For what I can see looking at this form history, it has always been like that. So, using the form in a default AtoM install the only languages that could have been added are those matching one of the cultures. I'm not completely sure at the moment (and the "ca@valencia" culture confuses me a bit) but I guess the translations workflow uses the culture codes and not the language codes and that you can't find a 3 chars language code in Weblate (or the old Transifex). So, as you said, we shouldn't have to worry about those languages.

#5 Updated by Dan Gillean over 1 year ago

Ok, good to know. In that case, then yes, let's see how much work you think will be required to update the select to use the culture list, not the language list. And - if a fix is sponsored (or we proceed internally), then i will need an updated list to share w users. Thanks Radda!

#6 Updated by José Raddaoui Marín over 1 year ago

Thanks Dan!

So, the list of cultures in Symfony I previously shared:

https://gist.github.com/jraddaoui/d7bf65f0b6b6c89054be3667e7b7b2af

As mentioned, using the culture validator from Symfony some of those don't pass, leaving the following list (with display names as requested):

https://gist.github.com/jraddaoui/f66126c189d4367f5e2d82e2533610ed

This is the code I used to generate the last list:

foreach (sfCultureInfo::getCultures() as $culture)
{
  if (sfCultureInfo::validCulture($culture))
  {
    print($culture.',"'.locale_get_display_name($culture).'"'.PHP_EOL);
  }
}

It requires the intl PHP extension to format the display name, so we may want to do it differently to avoid its installation (for example, splitting and using the i18n helpers from Symfony).

#8 Updated by Mike Cantelon over 1 year ago

  • Related to Bug #13222: Build SQL task changes the size of culture columns added

#9 Updated by David Juhasz 5 months ago

  • Related to Bug #13463: Problem: CSV import with pipe separated culture data crashes import and adds bad data added

#10 Updated by José Raddaoui Marín 4 months ago

  • Related to Bug #13465: Problem: users can add blank language rows in the i18n settings, causing indexing errors added

#11 Updated by Dan Gillean 4 months ago

  • Estimated time set to 12.00

#12 Updated by Dan Gillean 4 months ago

  • Estimated time changed from 12.00 to 16.00

Also available in: Atom PDF