[Bug 24211] New: Compress/uncompress translation files
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Bug ID: 24211 Summary: Compress/uncompress translation files Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: I18N/L10N Assignee: koha-bugs@lists.koha-community.org Reporter: bgkriegel@gmail.com QA Contact: testopia@bugs.koha-community.org CC: frederic@tamil.fr Translation files takes some space in an installed system, ~350 MB. It's not too much with current drives, but it's a space that can be saved on container images. Compressing them it's easy and can be made transparently, using only ~50 MB. Perhaps this enhancement could be used when generating packages. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Assignee|koha-bugs@lists.koha-commun |bgkriegel@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #1 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 96174 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96174&action=edit Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org, | |kyle@bywatersolutions.com, | |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #2 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice idea :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #96174|0 |1 is obsolete| | --- Comment #3 from David Nind <david@davidnind.com> --- Created attachment 96814 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96814&action=edit Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output) Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #96814|0 |1 is obsolete| | --- Comment #4 from Bouzid Fergani <bouzid.fergani@inlibro.com> --- Created attachment 98584 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98584&action=edit Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Bernardo, could you explain the different use cases here? Is the idea to commit the gz files? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #6 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- (In reply to Jonathan Druart from comment #5)
Bernardo, could you explain the different use cases here?
I see the following: 1) Compression before release (deb/tar.gz) In this case the installed system will use less space 2) Smaller docker images, perhaps using compressed files from deb/tar
Is the idea to commit the gz files?
I don't think so. I don't know much about git, but I think it would be worse for the size of the repository. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98584|0 |1 is obsolete| | --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 99375 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99375&action=edit Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #8 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Bernardo Gonzalez Kriegel from comment #6)
(In reply to Jonathan Druart from comment #5) 2) Smaller docker images, perhaps using compressed files from deb/tar
Is the idea to commit the gz files?
I don't think so. I don't know much about git, but I think it would be worse for the size of the repository.
We must not, that's why I was asking :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #9 from Julian Maurice <julian.maurice@biblibre.com> --- Have you tried to compile them (with msgfmt) instead of compressing them ? Maybe .mo files can be read by the translator tool without being uncompressed on disk first. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |In Discussion -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #10 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- (In reply to Julian Maurice from comment #9)
Have you tried to compile them (with msgfmt) instead of compressing them ? Maybe .mo files can be read by the translator tool without being uncompressed on disk first.
Current translation process does not use .mo files, we only use .po files as a bag of matching strings (msgid -> msgsrt) to do search and replace inspecting tt, xml and js files. Besides, .mo binary files are not much smaller than .po files :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #11 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Bernardo Gonzalez Kriegel from comment #10)
Current translation process does not use .mo files, we only use .po files as a bag of matching strings (msgid -> msgsrt) to do search and replace inspecting tt, xml and js files.
I know that. What I suggested was to use .mo files as bag of strings too. But that may require additional dependencies, so it may not be worth it, especially if:
.mo binary files are not much smaller than .po files :)
I just did a small test: msgfmt -o fr-FR-staff-prog.mo fr-FR-staff-prog.po gzip -k fr-FR-staff-prog.po du -h fr-FR-staff-prog* 824K fr-FR-staff-prog.mo 2,1M fr-FR-staff-prog.po 392K fr-FR-staff-prog.po.gz So, .mo files are smaller than .po files, but not as small as gzipped .po Which is kind of disappointing :/ Anyway, I just wanted to know if you had considered this approach. Thanks for answering. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #12 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com --- Comment #13 from Joy Nelson <joy@bywatersolutions.com> --- enhancement not backported to 19.11.x branch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | Text to go in the| |Allows to compress and release notes| |uncompress .po files used | |for translations using: | |./translate compress fr-FR | |./translate uncompress | |fr-FR -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I tried to describe this one, but I am not compeletely sure how it fits in with the other new translation features. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27235 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #15 from David Cook <dcook@prosentient.com.au> --- Interesting idea! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27236 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- I've opened Bug 27236 for the idea of packaging compressed PO files separately in a koha-i18n package... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24211 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- English trivia: I was thinking that uncompress wasn't a word and that it should be decompress. However, it appears that uncompress is a word! Decompress dates back to 1905 and uncompress dates back to 1987. It even looks like uncompress actually came into usage because of computer compression. (My spellchecker doesn't like uncompress, but it appears to be wrong.) https://www.merriam-webster.com/dictionary/decompress https://www.merriam-webster.com/dictionary/uncompress -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org