[Bug 18152] New: UNIMARC bib records imported with invalid 'a' char in label pos.9
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Bug ID: 18152 Summary: UNIMARC bib records imported with invalid 'a' char in label pos.9 Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: MARC Bibliographic record staging/import Assignee: gmcharlt@gmail.com Reporter: stephane.delaune@biblibre.com QA Contact: testopia@bugs.koha-community.org The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21) the problem is that C4::Charset::SetMarcUnicodeFlag (called in particular when we import a record) always add 'a' char in the 9th label'pos whereas it should do it just for MARC21 and NORMARC (not for UNIMARC) : C4::Charset::SetMarcUnicodeFlag add 'a' char in the 9th label character for MARC21 and NORMARC (it's normal), but just before doing this it call "$marc_record->encoding('UTF-8')" which is a MARC::Record function which, when called with 'UTF-8' parameter, do only one thing : add 'a' char in the 9th label character -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #1 from stephane.delaune@biblibre.com --- Created attachment 60547 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=60547&action=edit Bug 18152 : fix unimarc label in SetMarcUnicodeFlag The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21) the problem is that C4::Charset::SetMarcUnicodeFlag (called in particular when we import a record) always add 'a' char in the 9th label'pos whereas it should do it just for MARC21 and NORMARC (not for UNIMARC) : C4::Charset::SetMarcUnicodeFlag add 'a' char in the 9th label character for MARC21 and NORMARC (it's normal), but just before doing this it call "$marc_record->encoding('UTF-8')" which is a MARC::Record function which, when called with 'UTF-8' parameter, do only one thing : add 'a' char in the 9th label character This patch only removes this incorrect function call, so, when we import a bib record in UNIMARC : it no longer adds erroneous character (this does not change anything for MARC21 and NORMARC because SetMarcUnicodeFlag explicitly adds 'a' char in the 9th label for them) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #2 from stephane.delaune@biblibre.com --- Created attachment 60548 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=60548&action=edit to test import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #60548|0 |1 is obsolete| | --- Comment #3 from stephane.delaune@biblibre.com --- Created attachment 60549 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=60549&action=edit marc21-with-a-char-in-label-pos9.mrc -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #4 from stephane.delaune@biblibre.com --- Created attachment 60550 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=60550&action=edit marc21-without-a-char-in-label-pos9.mrc -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #5 from stephane.delaune@biblibre.com --- Created attachment 60551 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=60551&action=edit unimarc-without-a-char-in-label-pos9.mrc -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #6 from stephane.delaune@biblibre.com --- to test : before applying the patch : - change marcflavour syspref in UNIMARC - import unimarc-without-a-char-in-label-pos9.mrc in koha (see below how to): a 'a' char will be add in label pos.9 (it's the bug) - change marcflavour syspref in MARC21 - import marc21-with-a-char-in-label-pos9.mrc (record already have a 'a' char in label pos.9)in koha : nothing is change, label keep 'a' char in label pos.9 (it's correct for MARC21 records) - import marc21-without-a-char-in-label-pos9.mrc in koha : a 'a' char will be add in label pos.9 (it's correct for MARC21 records) - apply patch - change marcflavour syspref in UNIMARC - import unimarc-without-a-char-in-label-pos9.mrc in koha (see below how to): label pos.9 is blank (bug fixed \o/) - change marcflavour syspref in MARC21 - import marc21-with-a-char-in-label-pos9.mrc (record already have a 'a' char in label pos.9)in koha : nothing is change, label keep 'a' char in label pos.9 (it's correct for MARC21 records, It's like before the patch) - import marc21-without-a-char-in-label-pos9.mrc in koha : a 'a' char will be add in label pos.9 (like before the patch, it's correct for MARC21 records) Note : how to import a record in Koha : Home › Tools › Stage MARC records for import, find file, click on "upload",verify character encoding is UTF-8, click on "stage for import" ; Home › Tools › Manage staged MARC records : choose your file and clic on "import" button ; then you can click below on file name to see it (and note (or not) the presence of char 'a' in label pos.9) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |P1 - high Status|NEW |Needs Signoff Severity|enhancement |major -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Hello Stéphane, Maybe a dumb question: what side-effects this bug can cause? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |stephane.delaune@biblibre.c | |om -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.poulain@biblibre.com --- Comment #8 from Paul Poulain <paul.poulain@biblibre.com> --- Hi Jonathan, In fact... I don't know. That's invalid and being invalid is evil. more info : We're using the exporter to send the iso2709 file to another (proprietary) ILS, and the file is rejected, as it is invalid. "La loi est dure mais c'est la loi" ;-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Ok! So I let you find a signoffer at the hackfest :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #60547|0 |1 is obsolete| | --- Comment #10 from Alex Buckley <alexbuckley@catalyst.net.nz> --- Created attachment 62246 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=62246&action=edit Bug 18152 : fix unimarc label in SetMarcUnicodeFlag The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21) the problem is that C4::Charset::SetMarcUnicodeFlag (called in particular when we import a record) always add 'a' char in the 9th label'pos whereas it should do it just for MARC21 and NORMARC (not for UNIMARC) : C4::Charset::SetMarcUnicodeFlag add 'a' char in the 9th label character for MARC21 and NORMARC (it's normal), but just before doing this it call "$marc_record->encoding('UTF-8')" which is a MARC::Record function which, when called with 'UTF-8' parameter, do only one thing : add 'a' char in the 9th label character This patch only removes this incorrect function call, so, when we import a bib record in UNIMARC : it no longer adds erroneous character (this does not change anything for MARC21 and NORMARC because SetMarcUnicodeFlag explicitly adds 'a' char in the 9th label for them) Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Additional questions: 1. MARC::Record does not handle UNIMARC records correctly, so we should never call ->encoding('utf-8') from Koha, but SetMarcUnicodeFlag instead, right? 2. Sounds like the code in SetMarcUnicodeFlag should then be: If MARC21 or NORMARC: $marc_record->encoding('utf-8') else # unimarc do our stuffs 3. Please fix author's name of the patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #12 from stephane.delaune@biblibre.com --- Answers to the above questions : - Question 1 : I never said "MARC::Record does not handle UNIMARC records correctly" ! MARC::Record->encoding('utf-8') add 'a' char in the 9th label'pos, it's ok in MARC21 and NORMARC but not in UNIMARC (requires than the 9th character in labels must be blank) C4::Charset::SetMarcUnicodeFlag always call MARC::Record->encoding('utf-8') but it should only do for MARC21 and NORMARC - Question 2 : In SetMarcUnicodeFlag, if MARC21 or NORMARC already do «substr($leader, 9, 1) = 'a'» in addition to MARC::Record->encoding('utf-8') (Which does exactly the same thing for UNIMARC MARC21 NORMARC) We had the choice between : - move the «MARC::Record->encoding('utf-8')»'s call instead of «substr($leader, 9, 1) = 'a'» (it's your proposition) - delete the «MARC::Record->encoding('utf-8')»'s call and keep «substr($leader, 9, 1) = 'a'» (chosen solution) Both are equivalent because «MARC::Record->encoding('utf-8')» only do one thing : «substr($leader, 9, 1) = 'a'» - Question 3 : I do not understand the problem and I do not know how to do this -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #13 from Paul Poulain <paul.poulain@biblibre.com> --- Stephane, if you look at your patch:
From 9f6f0e05c9c44164d4c2d8f00f4bbd2afd87b847 Mon Sep 17 00:00:00 2001 From: Koha User <support@biblibre.com> <<<<<<< THIS IS WRONG Date: Wed, 22 Feb 2017 12:25:33 +0100
Set your name => https://wiki.koha-community.org/wiki/Version_Control_Using_Git#Set_your_name... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #62246|0 |1 is obsolete| | --- Comment #14 from stephane.delaune@biblibre.com --- Created attachment 63008 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63008&action=edit Bug 18152 : fix unimarc label in SetMarcUnicodeFlag only fix author's name of the patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #15 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to stephane.delaune from comment #12) "MARC::Record does not handle UNIMARC records correctly" and "MARC::Record->encoding('utf-8') add 'a' char in the 9th label'pos, it's ok in MARC21 and NORMARC but not in UNIMARC" mean the same thing to me -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |18533 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18533 [Bug 18533] MARC::Record->encoding does not handle UNIMARC records -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 63040 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63040&action=edit Bug 18152: Add tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 63041 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63041&action=edit Bug 18152: Leader 9 for UNIMARC records must be blank -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #18 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Paul, Stéphane, Regarding the description of this bug we are expecting the Leader 9 for UNIMARC records to be blank, does it make sense to set it in SetMarcUnicodeFlag? Let me know if you agree with the last patch, otherwise I will adapt the tests to make them pass without it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #19 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #18)
Regarding the description of this bug we are expecting the Leader 9 for UNIMARC records to be blank, does it make sense to set it in SetMarcUnicodeFlag?
I do not think it makes sense actually, must be done elsewhere. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #20 from stephane.delaune@biblibre.com --- (In reply to Jonathan Druart from comment #19)
(In reply to Jonathan Druart from comment #18)
Regarding the description of this bug we are expecting the Leader 9 for UNIMARC records to be blank, does it make sense to set it in SetMarcUnicodeFlag?
I do not think it makes sense actually, must be done elsewhere.
What does not make sense is that SetMarcUnicodeFlag adds an invalid character where it should not (which can be considered a bug). To do so elsewhere is to leave a bug in place to do corrective action elsewhere: a nonsense -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #21 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I do not understand your last comment. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #63008|0 |1 is obsolete| | Attachment #63040|0 |1 is obsolete| | Attachment #63041|0 |1 is obsolete| | --- Comment #22 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 63102 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63102&action=edit Bug 18152 : fix unimarc label in SetMarcUnicodeFlag The standard UNIMARC requires than the 9th character (starting from 0) in labels must be blank (while it may be 'a' in marc21) the problem is that C4::Charset::SetMarcUnicodeFlag (called in particular when we import a record) always add 'a' char in the 9th label'pos whereas it should do it just for MARC21 and NORMARC (not for UNIMARC) : C4::Charset::SetMarcUnicodeFlag add 'a' char in the 9th label character for MARC21 and NORMARC (it's normal), but just before doing this it call "$marc_record->encoding('UTF-8')" which is a MARC::Record function which, when called with 'UTF-8' parameter, do only one thing : add 'a' char in the 9th label character This patch only removes this incorrect function call, so, when we import a bib record in UNIMARC : it no longer adds erroneous character (this does not change anything for MARC21 and NORMARC because SetMarcUnicodeFlag explicitly adds 'a' char in the 9th label for them) Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #23 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 63103 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63103&action=edit Bug 18152: Add tests Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #24 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Seen with Stéphane: It does not make really sense to modify position 9 of leader for UNIMARC records in SetMarcUnicodeFlag, this flag is not related to encoding. I have opened bug 18533 to remove the MARC::Record->encoding calls everywhere and replace them with SetMarcUnicodeFlag. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |kyle@bywatersolutions.com --- Comment #25 from Kyle M Hall <kyle@bywatersolutions.com> --- Pushed to master for 17.05, thanks Stephane, Jonathan! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |katrin.fischer@bsz-bw.de --- Comment #26 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- These patches have been pushed to 16.11.x and will be in 16.11.08. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #27 from Julian Maurice <julian.maurice@biblibre.com> --- Pushed to 3.22.x for 3.22.21 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com --- Comment #28 from Mason James <mtj@kohaaloha.com> --- Pushed to 16.05.x, for 16.05.13 release -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=18910 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |18910 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18910 [Bug 18910] Regression: Z39.50 wrong conversion in Unimarc by Bug 18152 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #29 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- This patch introduced a regression, please provide a patch ASAP on bug 18910. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also|https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=18910 | CC| |fridolin.somers@biblibre.co | |m --- Comment #30 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #29)
This patch introduced a regression, please provide a patch ASAP on bug 18910.
I suggest to revert these 2 patches, Rmaints please take a look at bug 18910. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 acovaliov@gmail.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |acovaliov@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 --- Comment #31 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Was reverted from 16.11.x for the 16.11.10 release. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18152 Frederique Couffignal <fcouffignal@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fcouffignal@gmail.com -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org