[Bug 18331] New: Export csv in item search, exports all items in one line (see Bug 16914)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Bug ID: 18331 Summary: Export csv in item search, exports all items in one line (see Bug 16914) Change sponsored?: --- Product: Koha Version: 3.22 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: I18N/L10N Assignee: koha-bugs@lists.koha-community.org Reporter: katrin.fischer@bsz-bw.de QA Contact: testopia@bugs.koha-community.org CC: frederic@tamil.fr CSV exports of late orders and item search are broken again if you are using them from a 'complete' translation. The theory is that if the strings are not translated, then the file is only copied (preserves line breaks) and not translated (kills line breaks). Both late orders and item search csv expots are effected. For item search there is one string in itemsearch_item.csv.inc: %s %s %s %s "%s %svon %s%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s" Easiest way to test: - Install German - Test CSV export - it should be broken. - Look for the strings in itemsearch_item.csv in the staff po file and "untranslate" them. This is easiest done by adding "fuzzy". - Reinstall German - CSV shoudl be fixed -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |16914 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16914 [Bug 16914] Export csv in item search, exports all items in one line -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org Version|3.22 |master Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |jonathan.druart@bugs.koha-c |ity.org |ommunity.org -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 64239 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=64239&action=edit Bug 18331: Fix CSV export (once and for all!) I am going to try and explain the problem here in order to make it a have-to-read for next changes. There are several things to test when and most of the time we break something when we try to fix something else. You have to test different CSV export, not only one. There are not all processed the same way. For instance acqui/csv/basket.tt does not contain any strings to translate and the translate script will copy the original file as it. But reports/csv/orders_by_budget.tt will not! Indeed it contains "TOTAL" which will be translated. The generated tt file for the translated language will remove all the carriage returns! That's why we cannot rely on TT for newlines (contrary to bug 16914 assumption). There are two possible methods: 1/ Fix the translate script => Hum... nope 2/ Remove all carriage returns and make them explicit by including an file that only contains 1 carriage return This second method is implemented in this patch. How it works: Use the PRE_CHOMP "[%-" and POST_CHOMP "-%]" (http://www.template-toolkit.org/docs/manual/Config.html#section_PRE_CHOMP_PO...) to remove all the newlines that could be added by TT in the original (en) files. Then include the new_line.inc to add a new line. That way original and translated files will behave the same way. The BLOCK in the csv_headers avoid to have the newlines added, in any cases. For instance: by default we will have an empty line at the end of the headers, but the translated headers will not have it. Test plan: At least 2 signoffs will be needed, please test carefully! You will need to generate CSV with at least 2 entries! Test with the non-translated interface (en) and with the language you want. To test with a translated language you will *have to*: - cd misc/translator - perl translate update LANG - manually edit the po file, remove the fuzzy tags, correctly translate the string (do not forget the additional %s in the headers) - perl translate install LANG 1/ Export basket and basketgroup as CSV 2/ Export late orders as CSV 3/ Export items (from the item search) as CSV 4/ Export the 2 reports "Orders by fund" and "Cash register" as CSV -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Export csv in item search, |CSV exports need to be |exports all items in one |fixed once and for all |line (see Bug 16914) | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major --- Comment #2 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Upping severity a bit -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Guys, please try your hardest to test this patch, I am fed up with fixing this bug every release! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |18694 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18694 [Bug 18694] Translatability: Get rid of exposing [%% FOREACH in csv/cash_register_stats.tt -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Marc Véron <veron@veron.ch> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |veron@veron.ch --- Comment #4 from Marc Véron <veron@veron.ch> --- I first tested English CSV export of late orders with two items and got everything on one line. Just for testing, I - removed the chomps from line 3, changed to: [% INCLUDE empty_line.inc %] -> Fail, same result (one line) - added an empty line above line 3 (just for testing, it makes no sense with the intention of an empty line include) [%- USE KohaDates -%] [%- INCLUDE csv_headers/acqui/lateorders.tt -%] [% INCLUDE empty_line.inc %] [%- FOREACH o IN orders -%] -> Fail CSV-Export with header on one line, contents (both items on one line - additionally added an empty line above line above the second empty line include: (...)unt %]","[% o.claimed_date %]" [% INCLUDE empty_line.inc %] [%- END -%] ,,Total orders in late, [% orders.size %] -> CSV-Export OK, 1 header line, 2 lines for 2 orders, 1 line for Total Result: The trick with the empty line include did not to work. Then I reseted all, applied the patch again and added the following to the first and only line of koha-tmpl/intranet-tmpl/prog/en/includes/empty_line.inc [% '' %] Result: CSV Export worked as expected (for late orders, English) Then I tested with a translation and it worked as well. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I have retested and it works for me (late orders). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #6 from Marc Véron <veron@veron.ch> --- Re-tested on current master with patch applied, plack + memcached restarted, page reloaded, 2 late orders, CSV export, open in editor, result (in one line): ORDER DATE,ESTIMATED DELIVERY DATE,VENDOR,INFORMATION,TOTAL COST,BASKET,CLAIMS COUNT,CLAIMED DATE"2017-06-13 (0 days)","06/13/2017","PROBESHOP","Schokolade Author: Testmann, Hans.","1.00 x 1 = 1.00 (Fund 1)","Probebasket 1 (1)","0","""2017-06-13 (0 days)","06/13/2017","PROBESHOP","Republik im Fokus Author: Hanisch, Ernst.","2.00 x 1 = 2.00 (Fund 1)","Probebasket 1 (1)","0","",,Total orders in late, 2 If I add the [% '' %] to koha-tmpl/intranet-tmpl/prog/en/includes/empty_line.inc I get 4 lines (as expected): ORDER DATE,ESTIMATED DELIVERY DATE,VENDOR,INFORMATION,TOTAL COST,BASKET,CLAIMS COUNT,CLAIMED DATE "2017-06-13 (0 days)","06/13/2017","PROBESHOP","Schokolade Author: Testmann, Hans.","1.00 x 1 = 1.00 (Fund 1)","Probebasket 1 (1)","0","" "2017-06-13 (0 days)","06/13/2017","PROBESHOP","Republik im Fokus Author: Hanisch, Ernst.","2.00 x 1 = 2.00 (Fund 1)","Probebasket 1 (1)","0","" ,,Total orders in late, 2 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Did you update the po file correctly? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #8 from Marc Véron <veron@veron.ch> --- It happens in English (see comment #4) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I replaced empty_string with [% '' %] and I got the same result (i.e. it works). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #10 from Marc Véron <veron@veron.ch> --- (In reply to Jonathan Druart from comment #9)
I replaced empty_string with [% '' %] and I got the same result (i.e. it works).
...as in my tests from comment #6 What is the next step? - Change koha-tmpl/intranet-tmpl/prog/en/includes/empty_line.inc ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I'd like somebody else to test as we get different behaviours with the same code. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #12 from Marc Véron <veron@veron.ch> --- Hmm, there must be a difference. Just to exclude: Same version of Template Toolkit? About page: Template (2.22) 2.26 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #13 from Marc Véron <veron@veron.ch> --- Updated Template Toolkit to 2.27, same result. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #14 from Marc Véron <veron@veron.ch> --- I can fix and reproduce the issue with following steps: - Apply patch - restart plack / memcached with every step as appropriate - Verify that (on my installation) the CSV export is in one line - Edit empty_line.inc (using vim) - Set vim to display line breaks (:set list) - Add a new line after the existing one - Delete the first line - Save - Do a new CSV export -> Success - Revert all edits (using u in vim) - Save - Do a new CSV export -> Fail, export is again on one line -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi I tested this fresh from start to end and I am seeing no improvement with the patch: Tested on master: - translate update de-DE - translate install de-DE - Switch to de-DE: - item search export broken - acq basket export broken - late acq orders broken - late issues export - OK (uses CSV profiles) Tested with patch on master: - apply patch - restart_all - translate update de-DE - open file in poedit - check headers are translated and not fuzzy (orange in poedit) - item search for csv - seems unchanged? - acq basket was fuzzy - retranslated - late acq orders was fuzzy - retranslated - translate install de-DE - Test again: - item search - still broken - acq basket - still broken - late acq orders - still broken I am sorry, but this doesn't seem to be the fix we are looking for :( -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #16 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Ok, reading through the bug a bit more, I also tested with English templates. I am also seeing trouble with the patch applied: - item search - OK! - acq basket - still broken - late acq orders - still broken Without patch: - item search - OK - acq basket - OK - late acq orders - OK Sorry to have on better news. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Well, we have a problem here. Still works for me. Please follow carefully the following, there is something you are or I am missing. - Apply the patches - Restart plack and memcached # useless, but just in case... - perl translate update de-DE - Edit po/de-DE-staff-prog.po and modify the two following blocks (for the CSV content, then for the headers): #: intranet-tmpl/prog/en/modules/acqui/csv/lateorders.tt:1 msgid "" "%s %s %s %s \"%s (%s days)\",\"%s\",\"%s\",\"%s %sAuthor: %s.%s%sPublished " "by: %s.%s\",\"%s x %s = %s (%s)\",\"%s (%s)\",\"%s\",\"%s\" %s %s ,,Total " "orders in late, %s " msgstr "" "%s %s %s %s \"%s (%s Tage)\",\"%s\",\"%s\",\"%s %sVerfasser: %s.%s%sVeröffentlicht " "von: %s.%s\",\"%s x %s = %s (%s)\",\"%s (%s)\",\"%s\",\"%s\" %s %s ,,Anzahl " "verspäteter Bestellungen, %s " #: intranet-tmpl/prog/en/includes/csv_headers/acqui/lateorders.tt:1 msgid "" "%sORDER DATE,ESTIMATED DELIVERY DATE,VENDOR,INFORMATION,TOTAL COST,BASKET," "CLAIMS COUNT,CLAIMED DATE%s " msgstr "" "%sBestelldatum,Berechnetes Lieferdatum,Lieferant,Informaton,Gesamtkosten," "Bestellung,Anzahl Reklamationen,Reklamationsdatum%s " Make sure the fuzzy lines are removed! - perl translate install de-DE - Create 2 orders, close the basket - hit /cgi-bin/koha/acqui/lateorders.pl - Export the 2 orders (English template), I get: ORDER DATE,ESTIMATED DELIVERY DATE,VENDOR,INFORMATION,TOTAL COST,BASKET,CLAIMS COUNT,CLAIMED DATE "2017-07-10 (0 days)","07/13/2017","My Vendor","E Street shuffle : Author: Heylin, Clinton..","111.00 x 1 = 111.00 (Fund 1_2)","My Basket (1)","0","" "2017-07-10 (0 days)","07/13/2017","My Vendor","Introduction to Attic Greek / Author: Mastronarde, Donald J..Published by: University of California Press,.","22.00 x 1 = 22.00 (Main fund)","My Basket (1)","0","" ,,Total orders in late, 2 - Enable de-DE (pref language) - Export the 2 orders, I get: Bestelldatum,Berechnetes Lieferdatum,Lieferant,Informaton,Gesamtkosten,Bestellung,Anzahl Reklamationen,Reklamationsdatum "2017-07-10 (0 Tage)","07/13/2017","My Vendor","E Street shuffle : Verfasser: Heylin, Clinton..","111.00 x 1 = 111.00 (Fund 1_2)","My Basket (1)","0","" "2017-07-10 (0 Tage)","07/13/2017","My Vendor","Introduction to Attic Greek / Verfasser: Mastronarde, Donald J..Veröffentlicht von: University of California Press,.","22.00 x 1 = 22.00 (Main fund)","My Basket (1)","0","" ,,Anzahl verspäteter Bestellungen, 2 % wc /tmp/lateorders_en.csv 4 55 486 /tmp/lateorders_en.csv % wc /tmp/lateorders_de.csv 4 50 541 /tmp/lateorders_de.csv -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #18 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Jonathan, It's exactly what I did. I think the really curious thing is that for me it's broken the English templates too. So if we leave out all the po stuff - where could be the difference on this level? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #19 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- pmvers Template => 2.24 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #20 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Try to remove the content of template_cache_dir or disable it (and restart memcached + plack). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #21 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Not related with the current discussion, but found an error in the current patch: +++ b/koha-tmpl/intranet-tmpl/prog/en/modules/acqui/csv/basket.tt @@ -1,4 +1,6 @@ -[% INCLUDE csv_headers/acqui/basket.tt %] +[%- INCLUDE csv_headers/acqui/basket.tt -%] +[% INCLUDE empty_line.inc -%] last line should be +[%- INCLUDE empty_line.inc -%] -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #22 from Marc Véron <veron@veron.ch> --- Created attachment 65478 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65478&action=edit Bug 18331: Force tt to insert newline to empty_line.inc See end of comment #4 and IRC: http://irc.koha-community.org/koha/2017-08-03#i_1958062 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #23 from Lee Jamison <ldjamison@marywood.edu> --- Created attachment 65515 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65515&action=edit [FOLLOW-UP] Fix CSV export (once and for all!) Follow-up on Comment 21 to correct the empty_line.inc include in basket.tt -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Lee Jamison <ldjamison@marywood.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #64239|0 |1 is obsolete| | --- Comment #24 from Lee Jamison <ldjamison@marywood.edu> --- Created attachment 65581 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65581&action=edit Bug 18331: Fix CSV export (once and for all!) I am going to try and explain the problem here in order to make it a have-to-read for next changes. There are several things to test when and most of the time we break something when we try to fix something else. You have to test different CSV export, not only one. There are not all processed the same way. For instance acqui/csv/basket.tt does not contain any strings to translate and the translate script will copy the original file as it. But reports/csv/orders_by_budget.tt will not! Indeed it contains "TOTAL" which will be translated. The generated tt file for the translated language will remove all the carriage returns! That's why we cannot rely on TT for newlines (contrary to bug 16914 assumption). There are two possible methods: 1/ Fix the translate script => Hum... nope 2/ Remove all carriage returns and make them explicit by including an file that only contains 1 carriage return This second method is implemented in this patch. How it works: Use the PRE_CHOMP "[%-" and POST_CHOMP "-%]" (http://www.template-toolkit.org/docs/manual/Config.html#section_PRE_CHOMP_PO...) to remove all the newlines that could be added by TT in the original (en) files. Then include the new_line.inc to add a new line. That way original and translated files will behave the same way. The BLOCK in the csv_headers avoid to have the newlines added, in any cases. For instance: by default we will have an empty line at the end of the headers, but the translated headers will not have it. Test plan: At least 2 signoffs will be needed, please test carefully! You will need to generate CSV with at least 2 entries! Test with the non-translated interface (en) and with the language you want. To test with a translated language you will *have to*: - cd misc/translator - perl translate update LANG - manually edit the po file, remove the fuzzy tags, correctly translate the string (do not forget the additional %s in the headers) - perl translate install LANG 1/ Export basket and basketgroup as CSV 2/ Export late orders as CSV 3/ Export items (from the item search) as CSV 4/ Export the 2 reports "Orders by fund" and "Cash register" as CSV Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Leaving as Needs Signoff as 2 signoffs are required per Comment 1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Lee Jamison <ldjamison@marywood.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65478|0 |1 is obsolete| | --- Comment #25 from Lee Jamison <ldjamison@marywood.edu> --- Created attachment 65582 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65582&action=edit Bug 18331: Force tt to insert newline to empty_line.inc See end of comment #4 and IRC: http://irc.koha-community.org/koha/2017-08-03#i_1958062 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #26 from Lee Jamison <ldjamison@marywood.edu> --- Created attachment 65583 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65583&action=edit Fix CSV export (once and for all!) Follow-up on Comment 21 to correct the empty_line.inc include in basket.tt https://bugs.koha-community.org/show_bug.cgi?id=18331 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com Attachment #65515|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65581|0 |1 is obsolete| | Attachment #65582|0 |1 is obsolete| | Attachment #65583|0 |1 is obsolete| | --- Comment #27 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 65620 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65620&action=edit [SIGNED-OFF] Bug 18331: Fix CSV export (once and for all!) I am going to try and explain the problem here in order to make it a have-to-read for next changes. There are several things to test when and most of the time we break something when we try to fix something else. You have to test different CSV export, not only one. There are not all processed the same way. For instance acqui/csv/basket.tt does not contain any strings to translate and the translate script will copy the original file as it. But reports/csv/orders_by_budget.tt will not! Indeed it contains "TOTAL" which will be translated. The generated tt file for the translated language will remove all the carriage returns! That's why we cannot rely on TT for newlines (contrary to bug 16914 assumption). There are two possible methods: 1/ Fix the translate script => Hum... nope 2/ Remove all carriage returns and make them explicit by including an file that only contains 1 carriage return This second method is implemented in this patch. How it works: Use the PRE_CHOMP "[%-" and POST_CHOMP "-%]" (http://www.template-toolkit.org/docs/manual/Config.html#section_PRE_CHOMP_PO...) to remove all the newlines that could be added by TT in the original (en) files. Then include the new_line.inc to add a new line. That way original and translated files will behave the same way. The BLOCK in the csv_headers avoid to have the newlines added, in any cases. For instance: by default we will have an empty line at the end of the headers, but the translated headers will not have it. Test plan: At least 2 signoffs will be needed, please test carefully! You will need to generate CSV with at least 2 entries! Test with the non-translated interface (en) and with the language you want. To test with a translated language you will *have to*: - cd misc/translator - perl translate update LANG - manually edit the po file, remove the fuzzy tags, correctly translate the string (do not forget the additional %s in the headers) - perl translate install LANG 1/ Export basket and basketgroup as CSV 2/ Export late orders as CSV 3/ Export items (from the item search) as CSV 4/ Export the 2 reports "Orders by fund" and "Cash register" as CSV Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #28 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 65621 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65621&action=edit [SIGNED-OFF] Bug 18331: Force tt to insert newline to empty_line.inc See end of comment #4 and IRC: http://irc.koha-community.org/koha/2017-08-03#i_1958062 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #29 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 65622 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65622&action=edit [SIGNED-OFF] Bug 18331: Followup - Fix CSV export (once and for all!) Follow-up on Comment 21 to correct the empty_line.inc include in basket.tt https://bugs.koha-community.org/show_bug.cgi?id=18331 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #30 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 65623 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65623&action=edit [SIGNED-OFF] Bug 18331: Followup - fix path to orders_by_budget csv header Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #31 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Tested, each with English and German (translated): - Acquisitions: Basket export - OK! - Acquisitions: Basket group export - OK! - Acquisitions: Late orders - OK! - Item search - OK! - Report: Cash register stats - OK! (filed bug 19075 for improving header line) - Report: Orders by fund - OK! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65620|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65621|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65622|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65623|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #32 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 65815 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65815&action=edit [SIGNED OFF] Bug 18331: Fix CSV export (once and for all!) I am going to try and explain the problem here in order to make it a have-to-read for next changes. There are several things to test when and most of the time we break something when we try to fix something else. You have to test different CSV export, not only one. There are not all processed the same way. For instance acqui/csv/basket.tt does not contain any strings to translate and the translate script will copy the original file as it. But reports/csv/orders_by_budget.tt will not! Indeed it contains "TOTAL" which will be translated. The generated tt file for the translated language will remove all the carriage returns! That's why we cannot rely on TT for newlines (contrary to bug 16914 assumption). There are two possible methods: 1/ Fix the translate script => Hum... nope 2/ Remove all carriage returns and make them explicit by including an file that only contains 1 carriage return This second method is implemented in this patch. How it works: Use the PRE_CHOMP "[%-" and POST_CHOMP "-%]" (http://www.template-toolkit.org/docs/manual/Config.html#section_PRE_CHOMP_PO...) to remove all the newlines that could be added by TT in the original (en) files. Then include the new_line.inc to add a new line. That way original and translated files will behave the same way. The BLOCK in the csv_headers avoid to have the newlines added, in any cases. For instance: by default we will have an empty line at the end of the headers, but the translated headers will not have it. Test plan: At least 2 signoffs will be needed, please test carefully! You will need to generate CSV with at least 2 entries! Test with the non-translated interface (en) and with the language you want. To test with a translated language you will *have to*: - cd misc/translator - perl translate update LANG - manually edit the po file, remove the fuzzy tags, correctly translate the string (do not forget the additional %s in the headers) - perl translate install LANG 1/ Export basket and basketgroup as CSV 2/ Export late orders as CSV 3/ Export items (from the item search) as CSV 4/ Export the 2 reports "Orders by fund" and "Cash register" as CSV Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #33 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 65816 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65816&action=edit [SIGNED OFF] Bug 18331: Force tt to insert newline to empty_line.inc See end of comment #4 and IRC: http://irc.koha-community.org/koha/2017-08-03#i_1958062 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #34 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 65817 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65817&action=edit [SIGNED OFF] Bug 18331: Followup - Fix CSV export (once and for all!) Follow-up on Comment 21 to correct the empty_line.inc include in basket.tt https://bugs.koha-community.org/show_bug.cgi?id=18331 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #35 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 65818 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65818&action=edit [SIGNED OFF] Bug 18331: Followup - fix path to orders_by_budget csv header Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | CC| |m.de.rooy@rijksmuseum.nl --- Comment #36 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- The tt files now include: Header line [%- INCLUDE empty_line.inc -%] <= with final chomp Row information [%- INCLUDE empty_line.inc %] <= without final chomp But if I understand it correcty, this does not make a difference. I would recommend to add the final chomp to the second include too for consistency (all directives chomp at both sides and the empty line includes care for newlines). The newline is generated by processing the empty line include and that one is not chomped away anymore. Tested with basket.tt. Please let me know if I am mistaken. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #37 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- It would be great if we could get this one in before the 15th and string freeze. It's well tested now, please don't break it - there is a long history of 'argh' over this problem :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #38 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 65956 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65956&action=edit Bug 18331: POST_CHOMP everywhere! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #39 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Yes, basically the idea is to have PRE and POST CHOMP everywhere in the csv template files, then use the empty_line include file to add it explicitly. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65815|0 |1 is obsolete| | Attachment #65816|0 |1 is obsolete| | Attachment #65817|0 |1 is obsolete| | Attachment #65818|0 |1 is obsolete| | Attachment #65956|0 |1 is obsolete| | --- Comment #40 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 66011 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66011&action=edit Bug 18331: Fix CSV export (once and for all!) I am going to try and explain the problem here in order to make it a have-to-read for next changes. There are several things to test when and most of the time we break something when we try to fix something else. You have to test different CSV export, not only one. There are not all processed the same way. For instance acqui/csv/basket.tt does not contain any strings to translate and the translate script will copy the original file as it. But reports/csv/orders_by_budget.tt will not! Indeed it contains "TOTAL" which will be translated. The generated tt file for the translated language will remove all the carriage returns! That's why we cannot rely on TT for newlines (contrary to bug 16914 assumption). There are two possible methods: 1/ Fix the translate script => Hum... nope 2/ Remove all carriage returns and make them explicit by including an file that only contains 1 carriage return This second method is implemented in this patch. How it works: Use the PRE_CHOMP "[%-" and POST_CHOMP "-%]" (http://www.template-toolkit.org/docs/manual/Config.html#section_PRE_CHOMP_PO...) to remove all the newlines that could be added by TT in the original (en) files. Then include the new_line.inc to add a new line. That way original and translated files will behave the same way. The BLOCK in the csv_headers avoid to have the newlines added, in any cases. For instance: by default we will have an empty line at the end of the headers, but the translated headers will not have it. Test plan: At least 2 signoffs will be needed, please test carefully! You will need to generate CSV with at least 2 entries! Test with the non-translated interface (en) and with the language you want. To test with a translated language you will *have to*: - cd misc/translator - perl translate update LANG - manually edit the po file, remove the fuzzy tags, correctly translate the string (do not forget the additional %s in the headers) - perl translate install LANG 1/ Export basket and basketgroup as CSV 2/ Export late orders as CSV 3/ Export items (from the item search) as CSV 4/ Export the 2 reports "Orders by fund" and "Cash register" as CSV Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #41 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 66012 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66012&action=edit Bug 18331: Force tt to insert newline to empty_line.inc See end of comment #4 and IRC: http://irc.koha-community.org/koha/2017-08-03#i_1958062 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #42 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 66013 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66013&action=edit Bug 18331: Followup - Fix CSV export (once and for all!) Follow-up on Comment 21 to correct the empty_line.inc include in basket.tt https://bugs.koha-community.org/show_bug.cgi?id=18331 Signed-off-by: Lee Jamison <ldjamison@marywood.edu> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #43 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 66014 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66014&action=edit Bug 18331: Followup - fix path to orders_by_budget csv header Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #44 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 66015 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66015&action=edit Bug 18331: POST_CHOMP everywhere! Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #45 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 17.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable --- Comment #46 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Pushed to 17.05.x, will be in 17.05.04 Thanks for your job, a real never ending bug -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|CSV exports need to be |Translated CSV exports need |fixed once and for all |to be fixed once and for | |all -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18331 --- Comment #47 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- These patches have been pushed to 16.11.x and will be in 16.11.12. Let's hope this zombie bug stays dead now! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org