[Bug 18153] New: UNIMARC bib records exported with invalid 'a' char in label pos.9
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Bug ID: 18153 Summary: UNIMARC bib records exported with invalid 'a' char in label pos.9 Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Tools Assignee: gmcharlt@gmail.com Reporter: stephane.delaune@biblibre.com QA Contact: testopia@bugs.koha-community.org The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21), but under certain conditions (e.g. when a iso2709 file is imported) Koha's unimarc records can have an invalid char. 'a' in label pos.9 So when we want to export them with export's tool, if marcflavour syspref is UNIMARC, we need to delete this bad 'a' char from label pos.9 (to have blank) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #1 from stephane.delaune@biblibre.com --- Created attachment 60552 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=60552&action=edit Bug 18153 : fix unimarc label in export tool The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21), but under certain conditions (e.g. when a iso2709 file is imported) Koha's unimarc records can have an invalid char. 'a' in label pos.9 So when we want to export them with export's tool, if marcflavour syspref is UNIMARC, we need to delete this bad 'a' char from label pos.9 (to have blank) : this patch make this -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #2 from stephane.delaune@biblibre.com --- Created attachment 60553 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=60553&action=edit unimarc-withbad-a-char-in-label-pos9.mrc -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #3 from stephane.delaune@biblibre.com --- to test : before applying the patch : - change marcflavour syspref in UNIMARC - import unimarc-withbad-a-char-in-label-pos9.mrc in koha (see below how to) : this record have invalid char 'a' in label pos.9 - export your record (Home › Tools › Export data, choose your biblionumber in "from biblionumber:" and "To biblionumber:", Clear all Library) : open file in text editor and you will see 9th char (from0) is 'a' - apply patch - export your record (Home › Tools › Export data, choose your biblionumber in "from biblionumber:" and "To biblionumber:", Clear all Library) : open file in text editor and you will see 9th char (from0) is blank (bug is fixed) Note : how to import a record in Koha : Home › Tools › Stage MARC records for import, find file, click on "upload",verify character encoding is UTF-8, click on "stage for import" ; Home › Tools › Manage staged MARC records : choose your file and clic on "import" button ; then you can click below on file name to see it (and note (or not) the presence of char 'a' in label pos.9) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Priority|P5 - low |P1 - high Severity|enhancement |major -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #60552|0 |1 is obsolete| | --- Comment #4 from Alex Buckley <alexbuckley@catalyst.net.nz> --- Created attachment 62244 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=62244&action=edit Bug 18153 : fix unimarc label in export tool The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21), but under certain conditions (e.g. when a iso2709 file is imported) Koha's unimarc records can have an invalid char. 'a' in label pos.9 So when we want to export them with export's tool, if marcflavour syspref is UNIMARC, we need to delete this bad 'a' char from label pos.9 (to have blank) : this patch make this -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #62244|0 |1 is obsolete| | --- Comment #5 from Alex Buckley <alexbuckley@catalyst.net.nz> --- Created attachment 62245 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=62245&action=edit Bug 18153 : fix unimarc label in export tool The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21), but under certain conditions (e.g. when a iso2709 file is imported) Koha's unimarc records can have an invalid char. 'a' in label pos.9 So when we want to export them with export's tool, if marcflavour syspref is UNIMARC, we need to delete this bad 'a' char from label pos.9 (to have blank) : this patch make this Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #62245|0 |1 is obsolete| | --- Comment #6 from stephane.delaune@biblibre.com --- Created attachment 63009 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63009&action=edit Bug 18153 : fix unimarc label in export tool only fix author's name of the patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Must it be done when importing as well? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #8 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- The question is: Should not we call SetMarcUnicodeFlag from ModBiblioMarc to replace the whole "if ( $encoding eq "UNIMARC" ) {" block? (read s/encoding/marcflavour ...) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #9 from stephane.delaune@biblibre.com --- (In reply to Jonathan Druart from comment #8)
The question is: Should not we call SetMarcUnicodeFlag from ModBiblioMarc to replace the whole "if ( $encoding eq "UNIMARC" ) {" block?
(read s/encoding/marcflavour ...)
The purpose of this patch is not to fix the wrong call of «MARC::Record->encoding('utf-8')» to import (it's the goal of bug 18152), nor even everywhere (it's the goal of bug 18533) ; the goal is to fix exported records. Indeed, a lot of existing unimarc records in a lot of Koha are erroneous because of this bug, the goal is that they do not remain erroneous when exported. I don't see the relationship with ModBiblioMarc who has been corrected to not add the bad character, not to delete it (and export don't call ModBiblioMarc) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I have asked Julian to take a look. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #11 from Julian Maurice <julian.maurice@biblibre.com> --- QAing this now -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #12 from Julian Maurice <julian.maurice@biblibre.com> --- QA: 1. With the patch, the export tool also modifies the leader of exported authorities (position 9 is for authority type, so this is a problem) 2. The code is poorly formatted and is missing a comment to explain why we need to do this. 3. It only fixes exported records from the export tool and only for iso2709 format. If I export in marcxml or if I use the 'Save' button from the detail page, the leader is still invalid. 4. This patch states clearly that the leader may be invalid in Koha but tries to fix it at export-time instead of fixing it directly in Koha. I think it is possible to fix the leader at save-time, so that every biblio created/modified (whether it's by import or manually) will be saved with a correct leader. And maybe at read-time too, for already existing invalid records. @QAers, any opinions ? Failing QA for 1 and 2. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |stephane.delaune@biblibre.c | |om -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #13 from stephane.delaune@biblibre.com --- Regarding the fourth point: a patch to remove this invalid character (to fix the leader) at save-time records was developed with the bug 18152; however, while this bz was pushed master, it introduced a regression (bug 18910) that required a revert. Indeed, MARC::Record->encoding is initially planned for marc21 and therefore considers that the absence of this invalid character makes that it is not encoded in utf8 (even in unimarc). Koha often uses this erroneous (in UNIMARC) function so, waiting to have a viable alternative, it is best to keep this invalid character for the moment in Koha -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #63009|0 |1 is obsolete| | --- Comment #14 from stephane.delaune@biblibre.com --- Created attachment 68532 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68532&action=edit Bug 18153 : fix unimarc label in export tool new patch for QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtompset@hotmail.com Status|Needs Signoff |In Discussion --- Comment #15 from M. Tompsett <mtompset@hotmail.com> --- (In reply to Julian Maurice from comment #12)
1. With the patch, the export tool also modifies the leader of exported authorities (position 9 is for authority type, so this is a problem) 2. The code is poorly formatted and is missing a comment to explain why we need to do this.
The new patch seems to address this, but I haven't tested it yet.
3. It only fixes exported records from the export tool and only for iso2709 format. If I export in marcxml or if I use the 'Save' button from the detail page, the leader is still invalid.
I think this is a valid reason to keep it Failed QA, though I'm changing it to in discussion. The summary doesn't say only iso2709 export format. It suggests UNIMARC records in general.
4. This patch states clearly that the leader may be invalid in Koha but tries to fix it at export-time instead of fixing it directly in Koha. I think it is possible to fix the leader at save-time, so that every biblio created/modified (whether it's by import or manually) will be saved with a correct leader. And maybe at read-time too, for already existing invalid records. @QAers, any opinions ?
While I am not a QA person, I think this is a really good point. The leader record should be saved correctly in the first place, not need correction on export. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #16 from stephane.delaune@biblibre.com ---
4. This patch states clearly that the leader may be invalid in Koha but tries to fix it at export-time instead of fixing it directly in Koha. I think it is possible to fix the leader at save-time, so that every biblio created/modified (whether it's by import or manually) will be saved with a correct leader. And maybe at read-time too, for already existing invalid records. @QAers, any opinions ?
While I am not a QA person, I think this is a really good point. The leader record should be saved correctly in the first place, not need correction on export.
already answered, copy of my previous comment: Regarding the fourth point: a patch to remove this invalid character (to fix the leader) at save-time records was developed with the bug 18152; however, while this bz was pushed master, it introduced a regression (bug 18910) that required a revert. Indeed, MARC::Record->encoding is initially planned for marc21 and therefore considers that the absence of this invalid character makes that it is not encoded in utf8 (even in unimarc). Koha often uses this erroneous (in UNIMARC) function so, waiting to have a viable alternative, it is best to keep this invalid character for the moment in Koha -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=23542 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=18371 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #17 from Fridolin Somers <fridolin.somers@biblibre.com> --- Needs rebase because of Bug 24679 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18153 --- Comment #18 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Fridolin Somers from comment #17)
Needs rebase because of Bug 24679
Still applies. But I am not where this discussion is at... ? -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org