Problems with duplicate language translation keys / strings

[Cyboman]

Dear developers and tech enthusiasts

I have a very bad problem with language translation key duplicates.

These occur:

1. in IPS native apps
2. in 3rd party applications
3. in 3rd party plugins

I just wrote the following quote into the cookie notice support topic, but there are other applications affected too:

 

12 minutes ago, Cyboman said:

1. I'm using this plugin since the release days and I did many upgrades.
   As I'm translating all the plugins in a foreign language, I finally got the following problem approx 1 year ago:
   After another update, it created a few duplicate language strings...
   So f.e. now, I have the language key "cookieNotice_dismiss" twice.
   This creates another big problem: I had a former translation from "Got it!" to "OK" but after the duplication, the plugin always selects the NEW translation key, but the NEW translation key isn't translatable anymore, as it doesn't save, because an older translation key already exists.
   Please see the screenshot:

To say this again to explain the screenshot:

• key1 above is translated, but it isn't selected for showing, it always shows key2
• key2 there you can enter a translation, but it doesn't save what you enter there, as key1 translation is already above. I can't change key2! It's always empty content.
• Result: There is no translation possible! It's always showing the english version only!
• Solution: I must get rid of 1 of the keys, but nobody could help so far. Please help.

Currently I have these problems with duplicates in IP.Commerce (there I find the same language keys f.e. in IPS "System", so the key seems to have 2 related apps), in some legally required plugins etc. This strains my sales immensely.

So far, nobody was able to help me, but these duplicate keys mean a very big problem to me, as the affected keys don't translate at all anymore. It doesn't affect all keys, but a lot!

Please help me how I can get rid of these duplicate keys. I can't even imagine how this could happen!

Thanks.

[Daniel F]

You could run following query to identify all duplicate strings

SELECT *  FROM ibf_core_sys_lang_words where lang_id = 1 AND word_js = 0  group by word_key having count(*) >= 2

then, I would suggest to contact the third parties and tell them to remove the duplicates AND also to include an upgrade step to really remove the duplicate record from the core_sys_lang_words database table.

Removing it only from the lang.php file won't remove it for existing installations

 

I can't find any duplicates in ip.commerce and system. Could you tell us the keys? ( It's possible that some old installations have still some duplicates, so it would help if we could identify these strings to remove them   for them  )

[Cyboman]

Thanks, I will report the results of the query, but I know that my queries didn't succeed last time

The language translation duplicates exist in my installation in these apps:

• PHP & TXT Widget (all keys)
• In theme designs (if duplicating themes via copy or installing new version 4.1 -> 4.2)
• Cookie Notice (all keys)
• Points System (f.e. "donate" | "donation" | "donation_amount" is also used by Commerce)
• and_x_others (is as well in "System" as in "Blogs") and more...

The problem is the following:

1. When translation keys were already translated, and then they are duplicated (by whatever is causing this), these (former valid keys) lose their association with the according app/plugin/theme. The app/plugin/theme columns displays "None" - in all columns. (not sure if the 
2. The new (secondary) translation key is displaying the app/plugin/theme association correctly, BUT these new duplicate keys are not translatable. You can enter anything in there but it NEVER SAVES.
3. The IPS System only uses keys that are associated correctly. But they are empty! And there is no way to delete the keys without association (where the translation is fine) nor are the duplicates translatable.

Result: NO TRANSLATION POSSIBLE!

If anyone from the IPS staff is reading this topic, my IPS ticket numbers are:

#994695
#will add soon...

I have these problems on the "self hosted version" as well as in a "customer in the cloud (CiC)" installation
(I definitely won't be able to perform any database commands in the secured IPS cloud installation ) 

For example, I just tried to deinstall cookie notice.

Result:

1. it deinstalls the (untranslatable) keys that are associated with cookie notice correctly
2. it doesn't deinstall the (already translated) keys that were no longer associated with cookie notice unfortunately

I HAVE TO GET RID OF OLD / NOT ASSOCIATED KEYS IN MY TRANSLATION FILE ONE AFTER ONE!

Ideally, a tool would help that merges

new duplicate keys association name (=name of plugin/app/theme)
WITH
former key translation (=the translation I already did, but can't be used)

Thanks anyway

[Nathan Explosion]

What happens when you install a new plugin? Do the language strings become duplicated on a brand new plugin install? Not going to help with the clear-up but will give some idea of whether there is a bug to be looked at (in regards to cause)

As for how you got in to this situation - the only thing I can think of is that a database restore/merge has taken place at some point. Anything like that happen in the past?

 

[Cyboman]

5 hours ago, Nathan Explosion said:

What happens when you install a new plugin? Do the language strings become duplicated on a brand new plugin install? 

Yes, f.e. there is the key "donate" in IP.Commerce.

Then I installed the Points System App by Anatkin which uses the key "donate", too.

Result: now I have 2 assigned keys. One that is associated with IP.Commerce and another one that is associated with Points. As Points was installed later, it copied the key and the translation from IP.Commerce.

BUT: I can only translate the Points-Version, BUT: it always displays the Commerce-Version (that's not translatable anymore)

RESULT: I have to live with the former version. At least this one has already been translated. If it would not have been, I would never be able to translate and use it!

5 hours ago, Nathan Explosion said:

As for how you got in to this situation - the only thing I can think of is that a database restore/merge has taken place at some point. Anything like that happen in the past?

That's a very good question. I'm using IPS since ~10 years and we had some database backups restored But nothing merged. There were some foreign translations inside already done by a former admin. But the whole language database with all plugins were completely retranslated: 250 hours work! With bad problems rising now...

AND: I have another CiC mcommunity, that is ~half a year old, and there I installed a language pack from the IPS marketplace. After that I installed the above mentioned Points App, and the error occured, too! Example with the "donate" key given above.

BASICALLY I HAVE AN IMPORTANT MESSAGE FOR YOU DEVS TO AVOID MORE DAMAGE:

NEVER USE EXISTING LANGUAGE KEYS FOR YOUR APPS TWICE! ALWAYS CREATE UNIQUE KEYS!

AND BEYOND THAT, THERE IS ALSO SOMETHING WEIRD GOING ON ON TOP THAT OCCURS EVEN WITH REINSTALLATIONS OR UPDATES... (where 1 key will be dissociated and becomes "None" app/plugin/theme that can't be deleted by software yet) There is no function, no tool, and the app deinstallation only deinstalls the associated key.... One dangerous dissociated key persists forever...

Any further help / assistance is highly appreciated. Thanks.

[MADMAN32395]

11 hours ago, Daniel F said:

I can't find any duplicates in ip.commerce and system. Could you tell us the keys? ( It's possible that some old installations have still some duplicates, so it would help if we could identify these strings to remove them   for them  )

i've attached what I get when running the query you posted.

duplicate lang list.html

[Nathan Explosion]

Your additional information above explains the issue a lot more.

7 hours ago, Cyboman said:

BASICALLY I HAVE AN IMPORTANT MESSAGE FOR YOU DEVS TO AVOID MORE DAMAGE:

NEVER USE EXISTING LANGUAGE KEYS FOR YOUR APPS TWICE! ALWAYS CREATE UNIQUE KEY

If not already done so, you need to report this issue to each of the developers of any plugins/apps you find that produce this issue - don't rely on them popping in to this topic and seeing the information.

[Adriano Faria]

7 hours ago, Cyboman said:

NEVER USE EXISTING LANGUAGE KEYS FOR YOUR APPS TWICE! ALWAYS CREATE UNIQUE KEYS!

I use tons of existing lang bits on my apps, like DONATE on Crowdfunding. You can use them. You just can’t CREATE them in your lang.php, which is a totally different thing. 

[MADMAN32395]

8 hours ago, Adriano Faria said:

I use tons of existing lang bits on my apps, like DONATE on Crowdfunding. You can use them. You just can’t CREATE them in your lang.php, which is a totally different thing. 

I have two keys from pmviewer if ya want those Adriano. It's just not listed in the file I dropped for Daniel since he was looking for ips duplicates.

[Cyboman]

I think there a different circumstances for different users... we shouldn't look for the exact names, I f.e. have more than 100 for sure! There are also duplicates that originate from custom plugins. And others might have others too. F.e. I don't have the same as you have @MADMAN32395 but some of them  

9 hours ago, Adriano Faria said:

I use tons of existing lang bits on my apps, like DONATE on Crowdfunding. You can use them. You just can’t CREATE them in your lang.php, which is a totally different thing.

It's a different thing, yes! But I require my community to be translated completely. So if there are bad translation problems for important keys, this is a "I can't use it" argument in the worst case.

But you are right, there is no problem if you only use english language.

[jair101]

This can be solved if IPS checks the language strings of any third party app and does not allow duplicates. 

[Adriano Faria]

45 minutes ago, Cyboman said:

But you are right, there is no problem if you only use english language.

No, you misunderstood. Any dev can reuse any existing language, like DONATE, MEMBER, JOINED, etc. It will show its value from the the source app. What I cannot do is add those lang bits (DONATE, JOINED or MEMBER) in my lang.php, that will cause the issue when translating. 

[Cyboman]

Thanks so far for all the great explanations and I'm glad there are others out there too, that are in conflict with the same problem!

Probably it would be a good idea to just write a "TIPP" for the developers section, so that new developers directly can see

"Don't use IPS keys a second time in your lang.php. THIS WILL CREATE PROBLEMS WITH TRANSLATIONS!"

I will inform these developers to have them remove these language keys!

But unfortunately there is one problem left, as I have massive other duplicates, that are not associated with any app/plugin/theme anymore and I really don't know how to handle this problem. As mentioned above, an app deinstallation/update doesn't work for these "ALL 'NONE' KEYS"

[Cyboman]

Ok, today I discovered where the dissociated translation keys - at least some of them - come from:

1. Each app/plugin has an unique app/plugin name. This name isn't hardcoded in the app/plugin, but a language key like __app_xxxxx.
2. Usually I translate all app/plugin keys on a test installation (to test new apps for bugs), then export the translation xml (only the app/plugin xml, not the rest!) and import it to the live installation.
3. In the live installation, the system doesn't recognize, that it is the same app/plugin because of the translation! The untranslated version in the live system still uses the original english naming, but the translation xml is importing the translated naming...
4. What happens is: __app_xxxxx become a duplicate, the new key is associated, the former key (same key!) is dissociated. 
5. Problem: You can't remove dissociated keys anymore. An app deinstallation doesn't help either. Admins lose control over this key.

[Cyboman]

This problem hasn't been solved yet.

We really need a tool/plugin/IPS core function to delete orphaned (single or duplicate) language keys....

Please help.

I'm really running into bad problems if I can't translate f.e. a "cookie notice" and others that are a very important GDPR requirement. This is NOT a 3rd party developer problem, but an IPS software bugs problem that leads to legal complications.

[Daniel F]

It's IMO not up to IPS to fix 3rd party code and duplicate language strings.

We could of course include a tool which would remove duplicate strings, but they would get back once you update the 3rd party app or plugin and the coder didn't remove the duplicate string properly.

I've also sent you a private tool which I use to locate duplicates.

[DawPi]

59 minutes ago, Daniel F said:

I've also sent you a private tool which I use to locate duplicates.

Please be so kind and send to me too. ?

[Ramsesx]

1 hour ago, Daniel F said:

I've also sent you a private tool which I use to locate duplicates

Me too, or better add it here in or anywhere else on IPS, in the guides section for example. Thanks.

[Marius]

2 hours ago, Daniel F said:

I've also sent you a private tool which I use to locate duplicates.

Please, me too. I have the same problem. Thank you!

[Joel R]

5 hours ago, Daniel F said:

It's IMO not up to IPS to fix 3rd party code and duplicate language strings.

We could of course include a tool which would remove duplicate strings, but they would get back once you update the 3rd party app or plugin and the coder didn't remove the duplicate string properly.

I've also sent you a private tool which I use to locate duplicates.

Why don't you just post in this topic as attachment? 

[Adriano Faria]

https://invisioncommunity.com/files/category/131-ips-extras/

[CodingJungle]

2 hours ago, Joel R said:

Why don't you just post in this topic as attachment? 

https://invisioncommunity.com/files/file/8946-the-dev-toolbox/ ? dt code