https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #1 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 102432 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102432&action=edit Bug 25067: Move PO file manipulation code into gulp tasks misc/translator/translate was doing three different things: - extract translatable strings - create or update PO files - install translated templates This patch separates responsibilities by moving the string extraction code into several 'xgettext-like' scripts and adds gulp tasks to automate string extraction and PO files update This has several benefits: - gulp runs tasks in parallel, so it's a lot faster (updating all PO files is at least 10 times faster with my 4-cores CPU) - there is no need for $KOHA_CONF to be defined LangInstaller.pm relied on $KOHA_CONF to get the different paths needed. I'm not sure why, since string extraction and PO update should work on source files, not installed files - string extraction code can be more easily tested This patch also brings a couple of fixes and improvements: - TT string extraction (strings wrapped in [% t(...) %]) was done with Template::Parser and PPI, which was extremely slow, and had some problems (see bug 24797). This is now done with Locale::XGettext::TT2 (new dependency) which is a lot faster, and fixes bug 24797 - string extraction for system preferences was putting context inside msgid. For instance: # Accounting > Policy msgid "accounting.pref#FinePaymentAutoPopup# automatically display " "a print dialog for a payment receipt when making a payment.." Now context is put into msgctxt, and the reference is set, which is cleaner #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref msgctxt "Accounting > Policy > FinePaymentAutoPopup" msgid "automatically display a print dialog for a payment receipt " "when making a payment.." The downside is that some messages will have to be re-translated, especially short messages like 'Do', for which msgmerge has a hard time finding the corresponding new msgid. - Replace deprecated gulp-util by recommended minimist - Fix header in 3 PO files Test plan: 1. Run `yarn install` and install Locale::XGettext::TT2 2. Run `gulp po:update` 3. Verify the contents of updated PO files 4. Run `cd misc/translator && ./translate install <lang>` 5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated 6. Run `gulp po:create --lang <lang>` and verify that it created all PO files for that language 7. Run `prove t/misc/translator` -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I wonder about the pref file - the # comment kept all strings for one pref together and we have translated "Do" and "Don't" quite differently for some of them, according to context as it works not really well to put the Verb in front of the sentence in German like that. So it appearing more than once was a feature for us. Can we keep that? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Thx Julian. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED --- Comment #6 from Julian Maurice <julian.maurice@biblibre.com> --- Thanks Bernardo for this detailed analysis. It looks like I forgot some files in the gulp globs. In particular columns.def and the XSL files that are not related to a specific marc flavour. I will try to fix that ASAP. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff --- Comment #8 from Julian Maurice <julian.maurice@biblibre.com> --- With the last patch I get the same count of msgid than on master for opac-bootstrap and staff-prog -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #102432|0 |1 is obsolete| | --- Comment #10 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 103403 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103403&action=edit Bug 25067: Move PO file manipulation code into gulp tasks misc/translator/translate was doing three different things: - extract translatable strings - create or update PO files - install translated templates This patch separates responsibilities by moving the string extraction code into several 'xgettext-like' scripts and adds gulp tasks to automate string extraction and PO files update This has several benefits: - gulp runs tasks in parallel, so it's a lot faster (updating all PO files is at least 10 times faster with my 4-cores CPU) - there is no need for $KOHA_CONF to be defined LangInstaller.pm relied on $KOHA_CONF to get the different paths needed. I'm not sure why, since string extraction and PO update should work on source files, not installed files - string extraction code can be more easily tested This patch also brings a couple of fixes and improvements: - TT string extraction (strings wrapped in [% t(...) %]) was done with Template::Parser and PPI, which was extremely slow, and had some problems (see bug 24797). This is now done with Locale::XGettext::TT2 (new dependency) which is a lot faster, and fixes bug 24797 - string extraction for system preferences was putting context inside msgid. For instance: # Accounting > Policy msgid "accounting.pref#FinePaymentAutoPopup# automatically display " "a print dialog for a payment receipt when making a payment.." Now context is put into msgctxt, and the reference is set, which is cleaner #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref msgctxt "Accounting > Policy > FinePaymentAutoPopup" msgid "automatically display a print dialog for a payment receipt " "when making a payment.." The downside is that some messages will have to be re-translated, especially short messages like 'Do', for which msgmerge has a hard time finding the corresponding new msgid. - Replace deprecated gulp-util by recommended minimist - Fix header in 3 PO files Test plan: 1. Run `yarn install` and install Locale::XGettext::TT2 2. Run `gulp po:update` 3. Verify the contents of updated PO files 4. Run `cd misc/translator && ./translate install <lang>` 5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated 6. Run `gulp po:create --lang <lang>` and verify that it created all PO files for that language 7. Run `prove t/misc/translator` Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Need to install yarn & gulp, no errors -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |dependency --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Julian and Bernardo, happy to see the thorough testing of this change. If we could make koha-translate and the translate script behave the same with this change than before, I'd be in favor of it. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #15 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 103440 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103440&action=edit Bug 25067: Add translate create|update commands that call gulp -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #17 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Bernardo Gonzalez Kriegel from comment #16)

Sorry to bother you with this, but can we left the deprecation notice for verbose mode? I would like the 'same' behavior as before (It's my liking of course, and you can ignore it. I'll sign anyway)

In my opinion, people should be warned even if they don't want to, and they can use gulp directly if they don't want to see this warning, right ? :) But in the end, you are probably the only one directly concerned by this change, so if you think it's better this way, I'll make those changes ASAP -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #19 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 103558 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103558&action=edit Bug 25067: Hide deprecation notice unless --verbose flag is given -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103404|0 |1 is obsolete| | --- Comment #21 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 103563 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103563&action=edit Bug 25067: Add missing globs in gulpfile for po:extract Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103557|0 |1 is obsolete| | --- Comment #23 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 103565 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103565&action=edit Bug 25067: Add license notice to xgettext* scripts Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #25 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- All tested and signed! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|25566 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25566 [Bug 25566] Change in DecreaseLoanHighHolds behaviour -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103563|0 |1 is obsolete| | --- Comment #27 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 106504 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=106504&action=edit Bug 25067: Add missing globs in gulpfile for po:extract Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103565|0 |1 is obsolete| | --- Comment #29 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 106506 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=106506&action=edit Bug 25067: Add license notice to xgettext* scripts Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #31 from Julian Maurice <julian.maurice@biblibre.com> --- All patches rebased on master -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #33 from Alex Arnaud <alex.arnaud@biblibre.com> --- Also, i have concerns about the .pot generated. They are no longer cleaned now. Do we need these .pot files actually ? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #35 from Katrin Fischer <katrin.fischer@bsz-bw.de> ---

- In the test plan only installing a language is used with the old syntax, creating and updating are different. I think mixing would not be good. Why is it not possible to move all to the old syntax? and should we them move installing as well with the deprecation note you suggest?

Realizing that last bit made no sense: And should we not move installing to the new syntax as well with a deprecation note like you suggest? Overall I think we need to keep things as easy as possible since not only experienced Koha users will be using these commands (update, install, create). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #37 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 106807 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=106807&action=edit Bug 25067: Do not set msgctxt in pref PO files By having msgctxt set, msgmerge will "lose" some translations, forcing translators to re-translate them. Also, Pootle does not offer the ability to search by context, making it harder to translate all strings related to a single preference. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #39 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hm, I think I don't understand what you are referring to as source files. Will this work for tarball installations, packaging installations and git based installations alike? I believe we don't always have 2 copies of the files? I am worried this could break things for some of those? Adding the info in a comment could help a bit, I tihnk, but it could get quite a lot, maybe just the names of the prefs? I think it's a wrong assumption that update and create will only be used by our translation manager or developers. I believe libraries have for example changed terms in po files and similar for the normal GUI. But I'd be interested in your workflow for separate po for other use cases. :) -- You are receiving this mail because: You are watching all bug changes.

Hm, I think I don't understand what you are referring to as source files. Will this work for tarball installations, packaging installations and git based installations alike? I believe we don't always have 2 copies of the files? I am worried this could break things for some of those? By source files I mean the files that are committed in the git repo. All files

I think it's a wrong assumption that update and create will only be used by our translation manager or developers. I believe libraries have for example changed terms in po files and similar for the normal GUI. I think that we should not support nor encourage that. Even if it's "only"

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #41 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Katrin Fischer from comment #39) that you have directly after a git clone. IMO create and update should not be run on "installations", but directly on files inside the git repository, independently of any installation. 'create' and 'update' are essentially developer tools to manage PO files within the git repository. translations, it's still a fork of Koha. Everyone is free to do that, of course, but then they are on their own to maintain their changes. If translations are wrong, they should be fixed on translate.koha-community.org

But I'd be interested in your workflow for separate po for other use cases. :) Basically, for each client-specific development that adds new strings, we create a PO file in misc/translator/po/fr-FR-<category>-<ticket-id>.po containing all the new strings and their translation. When we need to update fr-FR templates we call a script that "concatenates" (msgcat) all fr-FR-<category>-*.po and runs misc/translator/translate install

-- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #106504|0 |1 is obsolete| | --- Comment #43 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 109273 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109273&action=edit Bug 25067: Add missing globs in gulpfile for po:extract Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #106506|0 |1 is obsolete| | --- Comment #45 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 109275 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109275&action=edit Bug 25067: Add license notice to xgettext* scripts Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #106807|0 |1 is obsolete| | --- Comment #47 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 109277 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109277&action=edit Bug 25067: Do not set msgctxt in pref PO files By having msgctxt set, msgmerge will "lose" some translations, forcing translators to re-translate them. Also, Pootle does not offer the ability to search by context, making it harder to translate all strings related to a single preference. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |mtj@kohaaloha.com --- Comment #49 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- 1. Run `yarn install` and install Locale::XGettext::TT2 It looks like Locale::XGettext::TT2 has no packages for Debian yet, can you confirm? Ended up installatin from CPAN. Adding Mason. 2. Run `gulp po:update` $ gulp po:update -bash: gulp: command not found I've tried following this: https://gulpjs.com/docs/en/getting-started/quick-start/ $ gulp po:update Finishes this time. 3. Verify the contents of updated PO files I have verified the contents of the de-DE files. Problem: It appears that a strings have jumped from one po file to another, requiring a lot of retranslations. Namely this seems to be the value_builder plugins that are no longer in the MARC flavor files but appear in staff now: Example string: twentyfour track This is now in the UNIMARC file, but with this patch appears in staff. Question: Reviewing the pref po looks like before, correct? 4. Run `cd misc/translator && ./translate install <lang>` Testing with de-DE again. Question: do we need a follow-up to change the package commands as well? Activated the language and the language switcher in the preferences. 5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated ! System preferences: remained English Staff GUI: OK OPAC GUI: OK XSLT in staff and OPAC: OK messages JS: OK (verified with OPAC cart strings) messages: OK (verified with "item" and "items" in staff result list) installer: OK (tested with es-ES) installed sample files: OK (tested with notices and frameworks) Preferences don't translate. Stopping here for now because of the problem with prefs. 6. Run `gulp po:create --lang <lang>` and verify that it created all PO files for that language 7. Run `prove t/misc/translator` Test Summary Report ------------------- t/misc/translator/xgettext-pref.t (Wstat: 3840 Tests: 16 Failed: 15) Failed tests: 1, 3-16 Non-zero exit status: 15 Files=3, Tests=56, 1 wallclock secs ( 0.03 usr 0.01 sys + 0.34 cusr 0.07 csys = 0.45 CPU) Result: FAIL -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #51 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 109296 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109296&action=edit Bug 25067: Fix globs for staff and opac string extraction It looks like gulp (or more likely its dependency 'glob') does not like negative ('!') globs and the 'nocase' option when used together We have to repeat the globs to get what we want (exclude all *MARC* files from the staff/opac string extraction process -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #53 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Katrin Fischer from comment #49)

1. Run `yarn install` and install Locale::XGettext::TT2

It looks like Locale::XGettext::TT2 has no packages for Debian yet, can you confirm? That's true. It is not packaged in Debian.

3. Verify the contents of updated PO files

I have verified the contents of the de-DE files.

Problem: It appears that a strings have jumped from one po file to another, requiring a lot of retranslations. Namely this seems to be the value_builder plugins that are no longer in the MARC flavor files but appear in staff now:

Example string: twentyfour track

This is now in the UNIMARC file, but with this patch appears in staff. Good catch! Thanks :) It seems that the update to gulp v4 has some issues with file globs and case-sensitivity. Fixed in one of the latest patches

I compared the number of msgid before and after the patch: Before: de-DE-installer-MARC21.po 1738 de-DE-installer.po 754 de-DE-marc-MARC21.po 3152 de-DE-marc-NORMARC.po 449 de-DE-marc-UNIMARC.po 1797 de-DE-messages-js.po 80 de-DE-messages.po 25 de-DE-opac-bootstrap.po 2014 de-DE-pref.po 2538 de-DE-staff-prog.po 9705 After: de-DE-installer-MARC21.po 1738 de-DE-installer.po 754 de-DE-marc-MARC21.po 3154 # 2 additional strings picked from koha-tmpl/intranet-tmpl/prog/en/data/marc21_field_CF.xsd, can be excluded if needed de-DE-marc-NORMARC.po 449 de-DE-marc-UNIMARC.po 1797 de-DE-messages-js.po 80 de-DE-messages.po 27 # 2 additional strings picked, thanks to the transition to Locale::XGettext::TT2 de-DE-opac-bootstrap.po 2014 de-DE-pref.po 2538 de-DE-staff-prog.po 9705

Question: Reviewing the pref po looks like before, correct? Yes, the pref po file content should not change (except for long lines wrapping, and maybe the order of strings ? I did not check)

4. Run `cd misc/translator && ./translate install <lang>`

Testing with de-DE again. Question: do we need a follow-up to change the package commands as well? I haven't tested those, but I don't think a change is needed

Activated the language and the language switcher in the preferences.

5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated

! System preferences: remained English Staff GUI: OK OPAC GUI: OK XSLT in staff and OPAC: OK messages JS: OK (verified with OPAC cart strings) messages: OK (verified with "item" and "items" in staff result list) installer: OK (tested with es-ES) installed sample files: OK (tested with notices and frameworks)

Preferences don't translate. Stopping here for now because of the problem with prefs. Fixed by the last patch

7. Run `prove t/misc/translator`

Test Summary Report ------------------- t/misc/translator/xgettext-pref.t (Wstat: 3840 Tests: 16 Failed: 15) Failed tests: 1, 3-16 Non-zero exit status: 15 Files=3, Tests=56, 1 wallclock secs ( 0.03 usr 0.01 sys + 0.34 cusr 0.07 csys = 0.45 CPU) Result: FAIL Fixed

Thank you Katrin for testing ! It's now ready to be tested again :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #109272|0 |1 is obsolete| | --- Comment #55 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 110011 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110011&action=edit Bug 25067: Move PO file manipulation code into gulp tasks misc/translator/translate was doing three different things: - extract translatable strings - create or update PO files - install translated templates This patch separates responsibilities by moving the string extraction code into several 'xgettext-like' scripts and adds gulp tasks to automate string extraction and PO files update This has several benefits: - gulp runs tasks in parallel, so it's a lot faster (updating all PO files is at least 10 times faster with my 4-cores CPU) - there is no need for $KOHA_CONF to be defined LangInstaller.pm relied on $KOHA_CONF to get the different paths needed. I'm not sure why, since string extraction and PO update should work on source files, not installed files - string extraction code can be more easily tested This patch also brings a couple of fixes and improvements: - TT string extraction (strings wrapped in [% t(...) %]) was done with Template::Parser and PPI, which was extremely slow, and had some problems (see bug 24797). This is now done with Locale::XGettext::TT2 (new dependency) which is a lot faster, and fixes bug 24797 - string extraction for system preferences was putting context inside msgid. For instance: # Accounting > Policy msgid "accounting.pref#FinePaymentAutoPopup# automatically display " "a print dialog for a payment receipt when making a payment.." Now context is put into msgctxt, and the reference is set, which is cleaner #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref msgctxt "Accounting > Policy > FinePaymentAutoPopup" msgid "automatically display a print dialog for a payment receipt " "when making a payment.." The downside is that some messages will have to be re-translated, especially short messages like 'Do', for which msgmerge has a hard time finding the corresponding new msgid. - Fix header in 4 PO files Test plan: 1. Run `yarn install` and install Locale::XGettext::TT2 2. Run `gulp po:update` 3. Verify the contents of updated PO files 4. Run `cd misc/translator && ./translate install <lang>` 5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated 6. Run `gulp po:create --lang <lang>` and verify that it created all PO files for that language 7. Run `prove t/misc/translator` Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Need to install yarn & gulp, no errors -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #109274|0 |1 is obsolete| | --- Comment #57 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 110013 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110013&action=edit Bug 25067: Add translate create|update commands that call gulp Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #109276|0 |1 is obsolete| | --- Comment #59 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 110015 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110015&action=edit Bug 25067: Hide deprecation notice unless --verbose flag is given Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #109295|0 |1 is obsolete| | --- Comment #61 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 110017 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110017&action=edit Bug 25067: Update tests for xgettext-pref -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #109297|0 |1 is obsolete| | --- Comment #63 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 110019 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110019&action=edit Bug 25067: Fix translation of system preferences files A previous patch reverted the string extraction mechanism for .pref files but did not update the 'install' part. This patch does just that. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #64 from Julian Maurice <julian.maurice@biblibre.com> --- Patches rebased on master -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #110011|0 |1 is obsolete| | Attachment #110012|0 |1 is obsolete| | Attachment #110013|0 |1 is obsolete| | Attachment #110014|0 |1 is obsolete| | Attachment #110015|0 |1 is obsolete| | Attachment #110016|0 |1 is obsolete| | Attachment #110017|0 |1 is obsolete| | Attachment #110018|0 |1 is obsolete| | Attachment #110019|0 |1 is obsolete| | --- Comment #66 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 110776 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110776&action=edit Bug 25067: Move PO file manipulation code into gulp tasks misc/translator/translate was doing three different things: - extract translatable strings - create or update PO files - install translated templates This patch separates responsibilities by moving the string extraction code into several 'xgettext-like' scripts and adds gulp tasks to automate string extraction and PO files update This has several benefits: - gulp runs tasks in parallel, so it's a lot faster (updating all PO files is at least 10 times faster with my 4-cores CPU) - there is no need for $KOHA_CONF to be defined LangInstaller.pm relied on $KOHA_CONF to get the different paths needed. I'm not sure why, since string extraction and PO update should work on source files, not installed files - string extraction code can be more easily tested This patch also brings a couple of fixes and improvements: - TT string extraction (strings wrapped in [% t(...) %]) was done with Template::Parser and PPI, which was extremely slow, and had some problems (see bug 24797). This is now done with Locale::XGettext::TT2 (new dependency) which is a lot faster, and fixes bug 24797 - Fix header in 4 PO files For backward compatibility, 'create' and 'update' commands of misc/translator/translate can still be used and will execute the corresponding gulp task Test plan: 1. Run `yarn install` and install Locale::XGettext::TT2 2. Run `gulp po:update` 3. Verify the contents of updated PO files 4. Run `cd misc/translator && ./translate install <lang>` 5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated 6. Run `gulp po:create --lang <lang>` and verify that it created all PO files for that language 7. Run `prove t/misc/translator` Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Need to install yarn & gulp, no errors -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |26547 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26547 [Bug 26547] Move context from msgid to msgctxt in pref PO files -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #110776|0 |1 is obsolete| | --- Comment #68 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 111642 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111642&action=edit Bug 25067: Move PO file manipulation code into gulp tasks misc/translator/translate was doing three different things: - extract translatable strings - create or update PO files - install translated templates This patch separates responsibilities by moving the string extraction code into several 'xgettext-like' scripts and adds gulp tasks to automate string extraction and PO files update This has several benefits: - gulp runs tasks in parallel, so it's a lot faster (updating all PO files is at least 10 times faster with my 4-cores CPU) - there is no need for $KOHA_CONF to be defined LangInstaller.pm relied on $KOHA_CONF to get the different paths needed. I'm not sure why, since string extraction and PO update should work on source files, not installed files - string extraction code can be more easily tested This patch also brings a couple of fixes and improvements: - TT string extraction (strings wrapped in [% t(...) %]) was done with Template::Parser and PPI, which was extremely slow, and had some problems (see bug 24797). This is now done with Locale::XGettext::TT2 (new dependency) which is a lot faster, and fixes bug 24797 - Fix header in 4 PO files For backward compatibility, 'create' and 'update' commands of misc/translator/translate can still be used and will execute the corresponding gulp task Test plan: 1. Run `yarn install` and install Locale::XGettext::TT2 2. Run `gulp po:update` 3. Verify the contents of updated PO files 4. Run `cd misc/translator && ./translate install <lang>` 5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated 6. Run `gulp po:create --lang <lang>` and verify that it created all PO files for that language 7. Run `prove t/misc/translator` Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Need to install yarn & gulp, no errors Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #70 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am passing QA here, but there are some caveats: 1) This adds a dependency on Locale::XGettext::TT2 that currently has no packages for Debian. 2) The second part that deals with the translations of system preferences on bug 26547 is not ready yet. I believe both patches should only be pushed together. I hope that moving this a step further will be a little push for things to be finished. The other option would be moving it into BLOCKED (leaving that to RM) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #72 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Julian Maurice from comment #71)

(In reply to Katrin Fischer from comment #70)

...

2) The second part that deals with the translations of system preferences on bug 26547 is not ready yet. I believe both patches should only be pushed together.

Why ? Translations of system preferences still work without bug 26547. I moved the sysprefs part to bug 26547 for the specific purpose of not blocking this one :-)

Maybe this is a misunderstanding then. I'll retest, but what shoudl the pref file look like with only this patch? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #74 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Mason, Could you have a look at how hard/easy Locale::XGettext::TT2 can be packaged (from 20.11 ideally)? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #76 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Julian Maurice from comment #75)

(In reply to Jonathan Druart from comment #74)

...

Mason, Could you have a look at how hard/easy Locale::XGettext::TT2 can be packaged (from 20.11 ideally)?

Should it be packaged ? It is a dev-only dependency, so it won't be used in a production environment. Still better to have it as a deb package, but not required IMO.

Debian packages provide a koha-translate command that remove then install for update. So we are good indeed. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #78 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- t/misc/translator/xgettext-tt2.t ................. Can't locate Locale/XGettext/TT2.pm in @INC (you may need to install the Locale::XGettext::TT2 module) (@INC contains: /kohadevbox/koha /kohadevbox/qa-test-tools /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl/5.28/parent.pm line 16. BEGIN failed--compilation aborted at /kohadevbox/koha/t/misc/translator/../../../misc/translator/xgettext-tt2 line 27. This will be a problem for Mason, if the non db dependent tests fail. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |20.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #81 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 113255 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113255&action=edit Bug 25067: Adjust paths in tests -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #83 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 113258 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113258&action=edit Bug 25067: Fix perlcritic errors -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #85 from Julian Maurice <julian.maurice@biblibre.com> --- I tried to go back to just before these patches to make xt/author/translatable-templates.t fail, unsuccessfully. I'm not sure what this script is supposed to test, because even with a completely broken template with invalid html and invalid TT markup, the tests pass. So it looks like the test is useless, and I suggest to remove it. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |26971 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26971 [Bug 26971] Remove obsolete test translatable-templates.t -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 --- Comment #88 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #81)

Created attachment 113255 [details] [review] Bug 25067: Adjust paths in tests

(In reply to Jonathan Druart from comment #83)

Created attachment 113258 [details] [review] Bug 25067: Fix perlcritic errors

Last 2 patches pushed to master for 20.11. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #89 from Lucas Gass <lucas@bywatersolutions.com> --- enhancement will not be backported to 20.05.x -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25067 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |28154 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28154 [Bug 28154] Translate script faces encoding issues -- You are receiving this mail because: You are watching all bug changes.