[Bug 17754] New: MARCXML encoding problems
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Bug ID: 17754 Summary: MARCXML encoding problems Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Tools Assignee: gmcharlt@gmail.com Reporter: mirko@abunchofthings.net QA Contact: testopia@bugs.koha-community.org Uploading authorities in MARCXML in 16.11 and master kills Umlauts while they work ok in the same file in mrc format. I have not checked biblios yet. Upload mrc: Universität Berlin (Freie Universität) Upload xml: Universitt̃ Berlin (Freie Universitt̃) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #1 from Mirko Tietgen <mirko@abunchofthings.net> --- Created attachment 58096 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58096&action=edit Minimal example mrc file -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #2 from Mirko Tietgen <mirko@abunchofthings.net> --- Created attachment 58097 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58097&action=edit Minimal example xml file -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Mirko Tietgen <mirko@abunchofthings.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|MARCXML encoding problems |MARCXML upload in Intranet | |is broken (encoding) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Mirko, I recreate the issue with your file, but I don't if I create an authority record with Koha (110$a Universität Berlin) and I upload them. Where does come from this record? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- auth-min.xml: XML 1.0 document, UTF-8 Unicode text from_koha.marcxml: XML 1.0 document, ASCII text -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=17842 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #5 from Mirko Tietgen <mirko@abunchofthings.net> --- Just had it again. The XML file is created with Catmandu. When I convert it to mrc (with Catmandu) it works. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #6 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- I can reproduce your problem too (with minimal authority.xml) The solution must be found in RecordsFromMARCXMLFile. Somehow the record format and binary encoding should be passed correctly to MARC::File::XML and friends. Currently, I do not see setting the record format. If it defaults to USMARC, this might already create an issue. And passing the encoding to $batch->next might not be correct as is done now. I tried some things, but they all failed ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #7 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- ping Galen: How should we set record format and binary encoding correctly here? Can't make it up from the cpan documentation. We cannot set the encoding already when we 'use' the module; we should do it later. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Actually the interface mixes format and encoding too. MARC-8 is not an encoding as the interface suggests ! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Why not? https://en.wikipedia.org/wiki/MARC-8 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #10 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Katrin Fischer from comment #9)
Why not? https://en.wikipedia.org/wiki/MARC-8
Well at least confusing :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #11 from David Cook <dcook@prosentient.com.au> --- Is this still an issue? I love character encoding issues, so I'm happy to examine people's data and try to reproduce (and fix) the problems. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=29333 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I just have run into this again. We have a MARCXML file that appears to be proper MARC. My editor tells me the encoding is UTF-8 and the umlauts show correctly. On import, the umlauts get broken. Also: we are strangely missing a 245$b where the first word contains an umlaut. If I convert it to .mrc with MARCEdit, which works without any issues or complaints, the file imports correctly. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|master |20.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #12)
I just have run into this again. We have a MARCXML file that appears to be proper MARC. My editor tells me the encoding is UTF-8 and the umlauts show correctly.
On import, the umlauts get broken. Also: we are strangely missing a 245$b where the first word contains an umlaut.
If I convert it to .mrc with MARCEdit, which works without any issues or complaints, the file imports correctly.
I had an issue like that once with a file that was using an older Chinese encoding. I tried converting it to UTF-8 with all kinds of tools and nothing produced made Koha happy. However, when I did the conversion in MARCEdit, I was able to import it into Koha with no problem. This was a while ago so I don't remember what the finer points were... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30996 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|MARCXML upload in Intranet |MARCXML upload in Intranet |is broken (encoding) |is broken (encoding) if LDR | |pos. 9 is missing --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think I solved the riddle: The problem appears, if LDR pos. 9 = a (encoding) is missing. The attached MARCXML record here is missing the leader and this position was unset in the records I had issues with. Once I fixed the records, the import worked as expected. I wonder if we could automatically assume UTF-8 encoding, if the encoding is not set? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com --- Comment #15 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- I was able to reproduce in current master (22.12.00.013). I tried Katrin's trick of editing manually the record in vim and it imported correctly. However, I only had one record in my file... Not sure it's reasonable to edit manually files with lots of records, especially the leader, it can be finnicky. I'm not even sure the leader I changed it to is correct... I will attach the record. Here's what I did: 1. Download attached file 2. Optionally, open the file in a text editor (I used vim) --> The encoding is specified in the first line <?xml version="1.0" encoding="UTF-8"?> --> Accents are displayed correctly 3. In the staff interface, go to Cataloging > Stage MARC records for import 4. Choose the file 5. Click Upload file 6. In the form, choose Format: MARCXML (leave the rest as is) 7. Click Stage for import 8. Click View batch --> In the table, accents are displayed correctly 9. Click on the title --> Accents are *not* displayed correctly 10. Click Import this batch into the catalog 11. Click Manage imported batch 12. Click View in the last column of the table --> Accents are *not* displayed correctly in the record details --> Accents are displayed correctly in the breadcrumbs, page/tab title 13. Click the MARC tab --> Accents are displayed correctly in the page header --> Accents are *not* displayed correctly in the record details (tab 2 for example) 14. Edit the file in a text editor (again I used vim) and replace the leader by 28152nam a2201417 i 4500 15. Try steps 3 to 13 again --> Accents are ok everywhere -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #16 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Created attachment 148663 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148663&action=edit MARCXML file with one record Attached record for my test plan is previous comment. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #17 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- (In reply to Katrin Fischer from comment #14)
I wonder if we could automatically assume UTF-8 encoding, if the encoding is not set?
We specify the encoding in the form when we stage the file. Should this override the encoding in the records? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 Olivier Hubert <olivier.hubert@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |olivier.hubert@inlibro.com --- Comment #18 from Olivier Hubert <olivier.hubert@inlibro.com> --- Looking at the source code for the Perl MARC::File::XML module that is used to read the XML file and build MARC records, the ninth character of the leader *must* be an 'a'. Otherwise, even if we pass the correct encoding to the MARC::Record::new_from_xml function, if the ninth character is not an 'a', the result will not be processed as UTF-8. I am no expert in MARCXML, but according to the schema (https://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd), each character of the leader must be present. I think that's why the MARC::File::XML module assumes that there will be either a space or an 'a' character at the ninth position. The possible solutions to this issue, as I see them, would be to: a) reject a file with an incorrect leader. b) modify the leader before importing to correct any existing mistake, although I don't see how we would be able to determine the missing character(s). c) find another module to import MARC records. d) become part of the team working on MARC::File::XML and change the source code :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17754 --- Comment #19 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Should we close this as wontfix? -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org