[Bug 27968] New: MARC framework CSV and ODS import incomplete or corrupted
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Bug ID: 27968 Summary: MARC framework CSV and ODS import incomplete or corrupted Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: System Administration Assignee: koha-bugs@lists.koha-community.org Reporter: victor@tuxayo.net QA Contact: testopia@bugs.koha-community.org CC: gmcharlt@gmail.com 1. CSV 1. Export default MARC framework in CSV 2. Create a new framework 3. import the CSV 4. See MARC structure of the new framework 5. View subfield of tag 240 6. Compare with default framework 7. Subfields are missing 2. ODS 1. Export default MARC framework in .ods 2. Open the .ods with LibreOffice 3. Save it 4. Create a new framework 5. import the .ods 6. See MARC structure of the new framework 7. View subfield of tag 240 8. Compare with default framework 9. See Also: 3, and Link:3 have appeared (the code of my new framework code is 3) For ODS, it seems a cell is offset and the framework code ends up in other columns. The raw ODS from Koha and the one saved by LibreOffice don't seem to differ on the 240 subfields. Both issues happen on master and on 19.11 (with bug 27569 applied) Not sure about severity. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27569 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |27569 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27569 [Bug 27569] marc-framework import function doesn't accept LibreOffice csv/ods file formats -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |jonathan.druart@bugs.koha-c |ity.org |ommunity.org Status|NEW |ASSIGNED Depends on|27569 | CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- [2021/03/17 07:47:02] [WARN] C4::ImportExportFramework::_processRow_DB(): DBI Exception: DBD::mysql::st execute failed: Incorrect integer value: '' for column 'isurl' at row 1 [for Statement "INSERT INTO marc_subfield_structure (tagfield,tagsubfield,liblibrarian,libopac,repeatable,mandatory,important,kohafield,tab,authorised_value,authtypecode,value_builder,isurl,hidden,frameworkcode,seealso,link,defaultvalue,maxlength,display_order) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE tagfield=?,tagsubfield=?,liblibrarian=?,libopac=?,repeatable=?,mandatory=?,important=?,kohafield=?,tab=?,authorised_value=?,authtypecode=?,value_builder=?,isurl=?,hidden=?,frameworkcode=?,seealso=?,link=?,defaultvalue=?,maxlength=?,display_order=?" with ParamValues: 0='999', 1='c', 2='Koha biblionumber', 3='Koha biblionumber', 4='0', 5='0', 6='0', 7='biblio.biblionumber', 8='-1', 9='', 10='', 11='', 12='', 13='-5', 14="X", 15='', 16='', 17='', 18='9999', 19='0', 20='999', 21='c', 22='Koha biblionumber', 23='Koha biblionumber', 24='0', 25='0', 26='0', 27='biblio.biblionumber', 28='-1', 29='', 30='', 31='', 32='', 33='-5', 34="X", 35='', 36='', 37='', 38='9999', 39='0'] at /kohadevbox/koha/C4/ImportExportFramework.pm line 986 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27569 [Bug 27569] marc-framework import function doesn't accept LibreOffice csv/ods file formats -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |17258 --- Comment #2 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Only happens when SQL strict mode is on. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17258 [Bug 17258] [OMNIBUS] MySQL 5.7 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 118357 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=118357&action=edit Bug 27968: Fix framework import if isurl has an incorrect value If isurl is set to undef in the DB, it will be exported as an empty string, which is an invalid value for isurl (int) Incorrect integer value: '' for column 'isurl' Test plan: Export framework structure in CSV and ODS, then reimport it and check that all the subfields are important correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- MariaDB [koha_kohadev]> select tagsubfield, isurl from marc_subfield_structure where tagfield=240 and frameworkcode="" limit 6; +-------------+-------+ | tagsubfield | isurl | +-------------+-------+ | 0 | NULL | | 1 | NULL | | 2 | NULL | | 6 | 0 | | 8 | 0 | | 9 | 0 | +-------------+-------+ 6 rows in set (0.01 sec) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #5 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Likely not related but, is that expected that 1. Create new framework 2. Marc structure 3. Internal server error CGI::Compile::ROOT::kohadevbox_koha_admin_marctagstructure_2epl::marctagstructure_2epl(): DBI Exception: DBD::mysql::st execute failed: 'koha_kohadev.biblio_framework.frameworktext' isn't in GROUP BY [for Statement "select count(*),marc_tag_structure.frameworkcode,frameworktext from marc_tag_structure,biblio_framework where biblio_framework.frameworkcode=marc_tag_structure.frameworkcode group by marc_tag_structure.frameworkcode"] at /usr/share/perl5/CGI/Emulate/PSGI.pm line 30 at /usr/share/perl5/DBIx/Class/Exception.pm line 77 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #6 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Found issue, test scenario 1 passes but not if I add a save with LibreOffice 1. Export default MARC framework in CSV 2. Open the .csv with LibreOffice and save it 3. Create a new framework 4. import the CSV 5. See MARC structure of the new framework 6. Tags are missing the trailing zeros (10 instead of 010) 7. View subfields of tag 10/010 8. Compare with default framework 9. Subfields are missing note, why does on the default framework, these two URLs don't yield the same result? /koha/admin/marc_subfields_structure.pl?tagfield=010&frameworkcode= /koha/admin/marc_subfields_structure.pl?tagfield=10&frameworkcode=3 It would also make sense that 10 doesn't yield anything. 10. Same with 031/31 default framework still has different output for 31 and 031 ODS issue when saved by LibreOffice is still there 1. Export default MARC framework in .ods 2. Open the .ods with LibreOffice 3. Save it 4. Create a new framework 5. import the .ods 6. See MARC structure of the new framework 7. View subfield of tag 010 8. Compare with default framework 9. See Also: 4, and Link:4 have appeared (the code of my new framework code is 4) Testing notes: Tags with a lot or diverse subfields: 010 100 240 995 031 110 600 700 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Victor Grousset/tuxayo from comment #6)
Found issue, test scenario 1 passes but not if I add a save with LibreOffice
1. Export default MARC framework in CSV 2. Open the .csv with LibreOffice and save it 3. Create a new framework 4. import the CSV 5. See MARC structure of the new framework 6. Tags are missing the trailing zeros (10 instead of 010)
I don't recreate that. Certainly a setting in the libreoffice config.
7. View subfields of tag 10/010 8. Compare with default framework 9. Subfields are missing note, why does on the default framework, these two URLs don't yield the same result? /koha/admin/marc_subfields_structure.pl?tagfield=010&frameworkcode= /koha/admin/marc_subfields_structure.pl?tagfield=10&frameworkcode=3 It would also make sense that 10 doesn't yield anything. 10. Same with 031/31 default framework still has different output for 31 and 031
10 picked the 100's subfields, that's definitely a bug, should be reported separately.
ODS issue when saved by LibreOffice is still there
1. Export default MARC framework in .ods 2. Open the .ods with LibreOffice 3. Save it 4. Create a new framework 5. import the .ods 6. See MARC structure of the new framework 7. View subfield of tag 010 8. Compare with default framework 9. See Also: 4, and Link:4 have appeared (the code of my new framework code is 4)
Does it work if you don't open/save it? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27989 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |27569 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27569 [Bug 27569] marc-framework import function doesn't accept LibreOffice csv/ods file formats -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #8 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Jonathan Druart from comment #7)
(In reply to Victor Grousset/tuxayo from comment #6)
1. Export default MARC framework in CSV 2. Open the .csv with LibreOffice and save it 3. Create a new framework 4. import the CSV 5. See MARC structure of the new framework 6. Tags are missing the trailing zeros (10 instead of 010)
I don't recreate that. Certainly a setting in the libreoffice config.
Next is attached the CSV saved by LibreOffice, 010 => 10 happens there. When opening the CSV, one of the options is "Format quoted fields as text.", it's off by default. Hence 010 => 10. Should the Koha import handle the spreadsheet software removing the leading zeros? Since one of the common spreadsheet software does this by default, a fair share of usage of this feature will be with this data.
7. View subfields of tag 10/010 8. Compare with default framework 9. Subfields are missing note, why does on the default framework, these two URLs don't yield the same result? /koha/admin/marc_subfields_structure.pl?tagfield=010&frameworkcode= /koha/admin/marc_subfields_structure.pl?tagfield=10&frameworkcode=3 It would also make sense that 10 doesn't yield anything. 10. Same with 031/31 default framework still has different output for 31 and 031
10 picked the 100's subfields, that's definitely a bug, should be reported separately.
Ok, see bug 27989. About the subfields missing it's related to the above issue. Because when reexporting they are in the CSV even if I can't find a way to see them in the interface.
ODS issue when saved by LibreOffice is still there
1. Export default MARC framework in .ods 2. Open the .ods with LibreOffice 3. Save it 4. Create a new framework 5. import the .ods 6. See MARC structure of the new framework 7. View subfield of tag 010 8. Compare with default framework 9. See Also: 4, and Link:4 have appeared (the code of my new framework code is 4)
Does it work if you don't open/save it?
Yes, if the ODS export is directly imported, no issues -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #9 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 118470 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=118470&action=edit Default-framework-export-resaved-libreoffice -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|27569 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27569 [Bug 27569] marc-framework import function doesn't accept LibreOffice csv/ods file formats -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #10 from Victor Grousset/tuxayo <victor@tuxayo.net> --- While there are still issues, this patch solves part of the problem and makes a few workflows work again. Shall the ticket be split for the remaining failing cases so the current patch can be merged? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Yes, I think we should go ahead with this patch and open a separate bug report for the other bug. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |28199 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28199 [Bug 28199] MARC framework CSV and ODS import incomplete or corrupted - part 2 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #118357|0 |1 is obsolete| | --- Comment #12 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 120036 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120036&action=edit Bug 27968: Fix framework import if isurl has an incorrect value If isurl is set to undef in the DB, it will be exported as an empty string, which is an invalid value for isurl (int) Incorrect integer value: '' for column 'isurl' Test plan: Export framework structure in CSV and ODS, then reimport it and check that all the subfields are important correctly Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #13 from Victor Grousset/tuxayo <victor@tuxayo.net> --- It works. Tested successfully CSV and ODS without going through LibreOffice. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl Status|Signed Off |BLOCKED --- Comment #14 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Looking here -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|BLOCKED |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #120036|0 |1 is obsolete| | --- Comment #15 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 120038 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120038&action=edit Bug 27968: Fix framework import if isurl has an incorrect value If isurl is set to undef in the DB, it will be exported as an empty string, which is an invalid value for isurl (int) Incorrect integer value: '' for column 'isurl' Test plan: Export framework structure in CSV and ODS, then reimport it and check that all the subfields are important correctly Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #16 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Passed QA Note that I would not call this a major bug. Devaluates severity ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|major |normal -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |21.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 21.05, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|21.05.00 |21.05.00,20.11.06 released in| | CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable --- Comment #18 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 20.11.x for 20.11.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Sarah Cornell <sbcornell@cityofportsmouth.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sbcornell@cityofportsmouth. | |com --- Comment #19 from Sarah Cornell <sbcornell@cityofportsmouth.com> --- What are our chances of having this pushed to 20.05? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable Version(s)|21.05.00,20.11.06 |21.05.00,20.11.06,20.05.12 released in| | --- Comment #20 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Pushed to 20.05.x for 20.05.12 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27968 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|21.05.00,20.11.06,20.05.12 |21.05.00,20.11.06,20.05.12, released in| |19.11.18 Status|Pushed to oldstable |Pushed to oldoldstable --- Comment #21 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Backported: Pushed to 19.11.x branch for 19.11.18 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org