[Bug 34467] New: OAI GetRecord bad encoding for Unimarc
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Bug ID: 34467 Summary: OAI GetRecord bad encoding for Unimarc Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Web services Assignee: koha-bugs@lists.koha-community.org Reporter: frederic@tamil.fr QA Contact: testopia@bugs.koha-community.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #1 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 154215 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=154215&action=edit Bug 34467: OAI GetRecord bad encoding for Unimarc For some Unimarc biblio records, the OAI Server returns a bad XML document, ie a document with extended characters, such as accents, not encoded in UTF8. It seems to come from somewhere in MARC::Record library trying to deduce biblio record endoding from some fields. It may work for MARC21 records. It doesn't for Unimarc records. So it's necessary to send to the method transforming record into XML the marcflavour. This way there is no encoding guessing from Unimarc biblio records content. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |frederic@tamil.fr |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #2 from Frédéric Demians <frederic@tamil.fr> --- The issue comes from leader position 9. In MARC21, if pos9 doesn't contain a 'a', MARC::File::XML transcode data from marc8 to utf8. In Unimarc, pos9 is not used (then often empty) and marc8 is never used. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|OAI GetRecord bad encoding |OAI GetRecord bad encoding |for Unimarc |for UNIMARC -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #3 from David Nind <david@davidnind.com> --- Are you able to add a test plan? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #4 from Frédéric Demians <frederic@tamil.fr> --- (In reply to David Nind from comment #3)
Are you able to add a test plan?
It seems possible, but, for me, now, the test t/db_dependent/OAI/Server.t fails on master. Is it the case for you also? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #5 from David Nind <david@davidnind.com> --- (In reply to Frédéric Demians from comment #4)
It seems possible, but, for me, now, the test t/db_dependent/OAI/Server.t fails on master. Is it the case for you also?
I can confirm that the tests fail for me, both before and after. I was more after a test plan, so I could replicate the issue, apply the patch, and make sure it fixes things. Test results - before ~~~~~~~~~~~~~~~~~~~~~ prove t/db_dependent/OAI/Server.t t/db_dependent/OAI/Server.t .. 13/34 # No tests run! t/db_dependent/OAI/Server.t .. 30/34 # Failed test 'No tests run for subtest "Bug 19725: OAI-PMH ListRecords and ListIdentifiers should use biblio_metadata.timestamp"' # at t/db_dependent/OAI/Server.t line 413. Can't call method "timestamp" on an undefined value at t/db_dependent/OAI/Server.t line 400. # Looks like your test exited with 2 just after 30. t/db_dependent/OAI/Server.t .. Dubious, test returned 2 (wstat 512, 0x200) Failed 5/34 subtests Test Summary Report ------------------- t/db_dependent/OAI/Server.t (Wstat: 512 Tests: 30 Failed: 1) Failed test: 30 Non-zero exit status: 2 Parse errors: Bad plan. You planned 34 tests but ran 30. Files=1, Tests=30, 3 wallclock secs ( 0.01 usr 0.00 sys + 1.12 cusr 0.13 csys = 1.26 CPU) Result: FAILU) Result: PASS Test results - after ~~~~~~~~~~~~~~~~~~~~ There was a lot more output, and I haven't included it here (just the start and end). prove t/db_dependent/OAI/Server.t t/db_dependent/OAI/Server.t .. 19/34 # Failed test 'ListRecords marcxml' # at t/db_dependent/OAI/Server.t line 184. # Compared $data->{"ListRecords"}{"record"}[0]{"metadata"}{"record"}{"leader"} # got : '00183 2200097 4500' # expect : '00183 a2200097 4500' # PARAM:--- # metadataPrefix: marcxml # verb: ListRecords ..... # request: http://localhost # xmlns: http://www.openarchives.org/OAI/2.0/ # xmlns:xsi: http://www.w3.org/2001/XMLSchema-instance # xsi:schemaLocation: http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd # No tests run! t/db_dependent/OAI/Server.t .. 30/34 # Failed test 'No tests run for subtest "Bug 19725: OAI-PMH ListRecords and ListIdentifiers should use biblio_metadata.timestamp"' # at t/db_dependent/OAI/Server.t line 413. Can't call method "timestamp" on an undefined value at t/db_dependent/OAI/Server.t line 400. # Looks like your test exited with 2 just after 30. t/db_dependent/OAI/Server.t .. Dubious, test returned 2 (wstat 512, 0x200) Failed 10/34 subtests Test Summary Report ------------------- t/db_dependent/OAI/Server.t (Wstat: 512 Tests: 30 Failed: 6) Failed tests: 21-24, 29-30 Non-zero exit status: 2 Parse errors: Bad plan. You planned 34 tests but ran 30. Files=1, Tests=30, 3 wallclock secs ( 0.02 usr 0.00 sys + 1.17 cusr 0.11 csys = 1.30 CPU) Result: FAIL -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #6 from Frédéric Demians <frederic@tamil.fr> --- (In reply to David Nind from comment #5)
I was more after a test plan, so I could replicate the issue, apply the patch, and make sure it fixes things.
On an UNIMARC Koha installation, modify let say biblio record 1234: - on leader position 9, enter a blank - on 200$a, add a è Get this biblio record from OAI server with GetRecord verb. Something like that: /cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=PREFIX:1234 Replace PREFIX with syspref OAI-PMH:archiveID. You will (should) get a 200$a without a è. Apply the patch, and repeat. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #7 from David Nind <david@davidnind.com> --- Created attachment 156583 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=156583&action=edit T2211: OAI GetRecord bad encoding for Unimarc For some Unimarc biblio records, the OAI Server returns a bad XML document, ie a document with extended characters, such as accents, not encoded in UTF8. It seems to come from somewhere in MARC::Record library trying to deduce biblio record endoding from some fields. It may work for MARC21 records. It doesn't for Unimarc records. So it's necessary to send to the method transforming record into XML the marcflavour. This way there is no encoding guessing from Unimarc biblio records content. https://bugs.koha-community.org/show_bug.cgi?id=34467 Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #8 from David Nind <david@davidnind.com> --- Thanks Frédéric! The information on how to test helped, and I have now signed it off. David Testing notes (using KTD): 1. Edited .env and change KOHA_MARC_FLAVOUR=unimarc 2. Enabled the OAI-PMH system preference. 3. Modified record 1234: - on leader position 9, enter a blank - on 200$a, add a è - Here is what is in the leader: 01552nam 2200829 4500 - Had to add all the required subfields - Edited the item for the record so that the item has the correct entries from item dropdown list options and the item type 4. Ran flush_memcached and restart_all and use a refreshed browser/ 5. Used this URL: http://127.0.0.1:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:1234 6. Note that the 200$a does not have an è. 7. Applied the patch, run flush_memcached and restart_all 8. The OAI response now includes an è in the 200$a -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #154215|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 156588 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=156588&action=edit Bug 34467: OAI GetRecord bad encoding for Unimarc For some Unimarc biblio records, the OAI Server returns a bad XML document, ie a document with extended characters, such as accents, not encoded in UTF8. It seems to come from somewhere in MARC::Record library trying to deduce biblio record endoding from some fields. It may work for MARC21 records. It doesn't for Unimarc records. So it's necessary to send to the method transforming record into XML the marcflavour. This way there is no encoding guessing from Unimarc biblio records content. https://bugs.koha-community.org/show_bug.cgi?id=34467 Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |23.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #10 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=34514 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #156583|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32336 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #11 from Fridolin Somers <fridolin.somers@biblibre.com> --- *** Bug 34515 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #12 from Fridolin Somers <fridolin.somers@biblibre.com> --- Great this is a duplicate of my Bug 34515 Maybe for another bug report, we should use $metadata->schema like did Bug 32336 : https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148628 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #13 from Fridolin Somers <fridolin.somers@biblibre.com> --- And please help me test Bug 34514 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Pushed to stable Version(s)|23.11.00 |23.11.00,23.05.05 released in| | --- Comment #14 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.05.x for 23.05.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |christophe.torin@univ-renne | |s2.fr --- Comment #15 from Fridolin Somers <fridolin.somers@biblibre.com> --- *** Bug 33994 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.11.00,23.05.05 |23.11.00,23.05.05,22.11.11 released in| | Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 --- Comment #16 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Nice work everyone! Pushed to oldstable for 22.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |36335 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36335 [Bug 36335] ILS-DI GetRecords bad encoding for UNIMARC -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |Needs documenting CC| |phil@chetcolibrary.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34467 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED --- Comment #17 from David Nind <david@davidnind.com> --- This fixes a bug. No changes required to the manual. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org