[Bug 11331] New: CSV export for viewlog.pl is missing newlines
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Bug ID: 11331 Summary: CSV export for viewlog.pl is missing newlines Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Tools Assignee: gmcharlt@gmail.com Reporter: sophie.meynieux@biblibre.com QA Contact: testopia@bugs.koha-community.org When you try to export the result of tools/viewlog.pl in csv, file cannot be correctly loaded : - Newline is missing after each record, - strings should be enclosed in "" - columns are not the same as for screen output This patch corrects this 3 points -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Sophie MEYNIEUX <sophie.meynieux@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |P4 Patch complexity|--- |Small patch Assignee|gmcharlt@gmail.com |sophie.meynieux@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 --- Comment #1 from Sophie MEYNIEUX <sophie.meynieux@biblibre.com> --- Created attachment 23265 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23265&action=edit Bug 11331 : corrects csv export for viewlog.pl (add newlines, string separator and same column as screen output) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Sophie MEYNIEUX <sophie.meynieux@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Christopher Brannon <cbrannon@cdalibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cbrannon@cdalibrary.org --- Comment #2 from Christopher Brannon <cbrannon@cdalibrary.org> --- Can you please provide a test plan? Christopher -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion CC| |dcook@prosentient.com.au --- Comment #3 from David Cook <dcook@prosentient.com.au> --- 1) I agree that a newline should be added. 2) I'm not certain about using double quotes as a text qualifier. While this will work for many of the logs, it won't work for any log that contains double quotes within it (such as a cataloguing (biblio or item) modification, authority modification, system preference modification). I think most programs typically use single quotes, double quotes, or spaces as text qualifiers, and all three of these won't work for us, since we have all three of those within our text. We also have commas in our text...and could possibly have tabs...so those two delimiters aren't very useful either without using text qualifiers. Of course, I don't see any easy solution to this problem. I think bibliographic data is too difficult to express in a regular delimited text format, and we're showing entire records in many of the logs. Perhaps we could clean problematic characters (tabs, commas, quotes, etc) from the logs, but then we also compromise the integrity of those logs. Perhaps another idea would be to export using XML formats (ODS,XLSX) instead. (We do this elsewhere in Koha.) Perhaps we should leave the simpler formats for simpler data. 3) "Title" and "author" should be removed from the list. You could probably safely do this without leaving the comment behind. -- Anyway, those are my thoughts. Since this patch doesn't completely fix the log export, I'm tempted to mark it as Failed QA. However, while it isn't a perfect fix, it is better than what we're currently doing, so maybe it's good enough for now. I'm marking it "In Discussion" to see what others think. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Christopher Brannon <cbrannon@cdalibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|cbrannon@cdalibrary.org | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 --- Comment #4 from Sophie MEYNIEUX <sophie.meynieux@biblibre.com> --- Is there a hope that logs could be exported one day ? What could be done to make this happen ? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolyn.somers@biblibre.co | |m Assignee|sophie.meynieux@biblibre.co |fridolyn.somers@biblibre.co |m |m --- Comment #5 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- I'm on a patch developpement -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23265|0 |1 is obsolete| | --- Comment #6 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- Created attachment 30264 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=30264&action=edit Bug 11331 - CSV export for viewlog.pl is missing newlines When you try to export the result of tools/viewlog.pl in csv, file cannot be correctly loaded : - newline is missing after each record, - strings should be enclosed in "" - columns are not the same as for screen output This patch corrects this by using like other export Text::CSV. Adds a header line made with the keys of first data. For that, all data values are initialiszed with empty string. Test plan : - Use a database with some logs, see sysprefs /cgi-bin/koha/admin/preferences.pl?tab=logs - Go to export page /cgi-bin/koha/tools/viewlog.pl - Select a module - Click on "To a file" and choose a file name - Click on "Submit" - Open file => Without this patch : newline is missing, multi-lines cells are not enclosed in "", there are no column headings => Without this patch : each line is a data line, complexe cells are enclosed in "", there are column headings - Test the export of all modules to see that all headings are necessary - Check the output to screen in the browser -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 --- Comment #7 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- Created attachment 30265 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=30265&action=edit Bug 11331 - CSV export for viewlog.pl is missing newlines - followup Perl formatting and cleaning. Also corrects the HTML of restults table in viewlog.tt. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |/cgi-bin/koha/tools/viewlog | |.pl -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #30264|0 |1 is obsolete| | --- Comment #8 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 30466 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=30466&action=edit [SIGNED-OFF] Bug 11331 - CSV export for viewlog.pl is missing newlines When you try to export the result of tools/viewlog.pl in csv, file cannot be correctly loaded : - newline is missing after each record, - strings should be enclosed in "" - columns are not the same as for screen output This patch corrects this by using like other export Text::CSV. Adds a header line made with the keys of first data. For that, all data values are initialiszed with empty string. Test plan : - Use a database with some logs, see sysprefs /cgi-bin/koha/admin/preferences.pl?tab=logs - Go to export page /cgi-bin/koha/tools/viewlog.pl - Select a module - Click on "To a file" and choose a file name - Click on "Submit" - Open file => Without this patch : newline is missing, multi-lines cells are not enclosed in "", there are no column headings => Without this patch : each line is a data line, complexe cells are enclosed in "", there are column headings - Test the export of all modules to see that all headings are necessary - Check the output to screen in the browser Signed-off-by: Owen Leonard <oleonard@myacpl.org> The CSV export is significantly improved. I question the usefulness of including biblioitemnumber in the output. A better inclusion would be itemnumber. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #30265|0 |1 is obsolete| | --- Comment #9 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 30467 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=30467&action=edit [SIGNED-OFF] Bug 11331 - CSV export for viewlog.pl is missing newlines - followup Perl formatting and cleaning. Also corrects the HTML of restults table in viewlog.tt. Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #30466|0 |1 is obsolete| | Attachment #30467|0 |1 is obsolete| | --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 30493 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=30493&action=edit [PASSED QA] Bug 11331 - CSV export for viewlog.pl is missing newlines When you try to export the result of tools/viewlog.pl in csv, file cannot be correctly loaded : - newline is missing after each record, - strings should be enclosed in "" - columns are not the same as for screen output This patch corrects this by using like other export Text::CSV. Adds a header line made with the keys of first data. For that, all data values are initialiszed with empty string. Test plan : - Use a database with some logs, see sysprefs /cgi-bin/koha/admin/preferences.pl?tab=logs - Go to export page /cgi-bin/koha/tools/viewlog.pl - Select a module - Click on "To a file" and choose a file name - Click on "Submit" - Open file => Without this patch : newline is missing, multi-lines cells are not enclosed in "", there are no column headings => Without this patch : each line is a data line, complexe cells are enclosed in "", there are column headings - Test the export of all modules to see that all headings are necessary - Check the output to screen in the browser Signed-off-by: Owen Leonard <oleonard@myacpl.org> The CSV export is significantly improved. I question the usefulness of including biblioitemnumber in the output. A better inclusion would be itemnumber. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> While this feature is still not perfect, this is a big improvement. Passes tests and QA script, restores basic functionality. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 30494 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=30494&action=edit [PASSED QA] Bug 11331 - CSV export for viewlog.pl is missing newlines - followup Perl formatting and cleaning. Also corrects the HTML of results table in viewlog.tt. Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |tomascohen@gmail.com --- Comment #12 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Patches pushed to master. Thanks Fridolin! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 --- Comment #13 from Jonathan Druart <jonathan.druart@biblibre.com> --- Comment on attachment 30493 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=30493 [PASSED QA] Bug 11331 - CSV export for viewlog.pl is missing newlines Review of attachment 30493: --> (http://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=11331&attachment=30493) ----------------------------------------------------------------- ::: tools/viewlog.pl @@ +201,5 @@
+ my $csv = Text::CSV::Encoded->new( { encoding_out => 'utf8', sep_char => $delimiter } ); + $csv or die "Text::CSV::Encoded->new FAILED: " . Text::CSV::Encoded->error_diag(); + # First line with heading + # Exporting bd id seems useless + my @headings = grep { $_ ne 'action_id' } sort keys $data[0];
Raises an error with perl <= 5.10 Type of arg 1 to keys must be hash (not array element) at /home/koha/src/tools/viewlog.pl line 214, near "];" -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |13522 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com Keywords| |rel_3_16_candidate -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |In Discussion --- Comment #14 from Mason James <mtj@kohaaloha.com> ---
Raises an error with perl <= 5.10
Type of arg 1 to keys must be hash (not array element) at /home/koha/src/tools/viewlog.pl line 214, near "];"
reopening bug to fix this new error -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 --- Comment #15 from Mason James <mtj@kohaaloha.com> --- (In reply to Mason James from comment #14)
Raises an error with perl <= 5.10
Type of arg 1 to keys must be hash (not array element) at /home/koha/src/tools/viewlog.pl line 214, near "];"
reopening bug to fix this new error
fyi: i think the following mod might fix the above fatal - my @headings = grep { $_ ne 'action_id' } sort keys $data[0]; + my @headings = grep { $_ ne 'action_id' } sort keys %{$data[0]}; -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Passed QA --- Comment #16 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- Bug 13522 is pushed to master and stable so I close discussion on this bug and set it back to pushed to master -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11331 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_3_16_candidate | --- Comment #17 from Mason James <mtj@kohaaloha.com> --- (In reply to Tomás Cohen Arazi from comment #12)
Patches pushed to master.
Pushed to 3.16.x, will be in 3.16.10 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org