[Bug 37184] New: Special character encoding problem when importing MARC file from the Acquisitions module
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Bug ID: 37184 Summary: Special character encoding problem when importing MARC file from the Acquisitions module Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Acquisitions Assignee: koha-bugs@lists.koha-community.org Reporter: alexandre.noel@inlibro.com QA Contact: testopia@bugs.koha-community.org Created attachment 168086 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168086&action=edit MARC file containing special characters To reproduce: 1. Go to Cataloging > "Stage records for import". 2. Upload the file 'ExportMemento2024061010532869Marc8.mrc'. 3. In the form, select the options : - Record type: Bibliographic - Character encoding: MARC8 - Format: MARC 4. Click "Stage for import". 5. Verify the titles of the records --> There should be no symbols or other signs of character encoding errors. 6. Find or create an open basket in the Acquisitions module. 7. Click "add to basket". 8. Select "From a staged file" 9. Select the staged file 10. Click "Select all" to check all records. 11. In the item information tab, choose the "Document type." 12. Verify the titles of the records in the basket: --> Koha replaces accents with symbols � -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Alexandre Noel <alexandre.noel@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alexandre.noel@inlibro.com, | |caroline.cyr-la-rose@inlibr | |o.com, | |philippe.blouin@inlibro.com --- Comment #1 from Alexandre Noel <alexandre.noel@inlibro.com> --- I think this bz needs more precision. The attached file has been exported in MARC-8, which is why we want to select MARC8 for character encoding in the import page (in test plan). By searching the code and the database, I found that Koha converts the imported file from MARC-8 to UTF-8, but in the 'import_records' table, in the database, the columns 'encoding' remains as MARC-8. In 'Koha::Import::Record.pm', it seems the statement : my $record = MARC::Record->new_from_xml($self->marcxml, $self->$encoding, $format); is converting a UTF-8 encoded file from MARC-8 to UTF-8 again, resulting in '�' symbols for special charaters. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Alexandre Noel <alexandre.noel@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 --- Comment #2 from Alexandre Noel <alexandre.noel@inlibro.com> --- Created attachment 168236 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168236&action=edit Bug 37184 - Special character encoding problem when importing MARC file from the Acquisitions module 1. Go to Cataloging > "Stage records for import". 2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file. 3. In the form, select the options : - Record type: Bibliographic - Character encoding: MARC8 - Format: MARC 4. Click "Stage for import". 5. Find or create an open basket in the Acquisitions module. 6. Click "add to basket". 7. Select "From a staged file" and select the previous staged file. 9. Click "Select all" to check all records. 10. In the item information tab, choose the Document type. 11. Verify the titles of the records in the basket: --> Koha replaces accents with symbols � 12. Apply the patch. 13. Do the same from step 1 and notice there is no more encoding issues. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl, | |phil@chetcolibrary.org --- Comment #3 from Phil Ringnalda <phil@chetcolibrary.org> --- That seems like an odd place to be making the change: if we took a MARC-8 record, converted it to UTF-8, and then stored it in a database table with an encoding column, why did we set that column to MARC-8 instead of UTF-8? And out of the five non-UTF-8 choices while staging, is MARC-8 the only one we convert to UTF-8, or would this need to just set UTF-8 for everything, ignoring $self->encoding? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Certainly sounds interesting but not sure about this fix -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 --- Comment #5 from Phil Ringnalda <phil@chetcolibrary.org> --- Oh, the encoding param in MARC::Record->new_from_xml is "what encoding do you want for the record I return?" out of a choice of UTF-8 or MARC-8. So setting $encoding to "UTF-8" if it's "MARC-8" is the same as just passing the literal string "UFT-8", which is what we do in almost every case elsewhere in the codebase, with some odd exceptions. C4::ImportBatch sometimes calls ->new_from_xml( StripNonXmlChars( $row->{'marcxml'} ), 'UTF-8', $row->{'encoding'} ) or ->new_from_xml( StripNonXmlChars($rowref->{'marcxml_old'}), 'UTF-8', $rowref->{'encoding'}, $marc_type); which seems a bit odd, not sure how passing two encodings to something that wants an encoding and a marc flavor is supposed to work. I'd be happy to sign off a patch that skips setting $encoding and just passes new_from_xml the string "UTF-8", and let QA worry about what on earth someone was thinking passing $self->encoding. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to Phil Ringnalda from comment #5)
Oh, the encoding param in MARC::Record->new_from_xml is "what encoding do you want for the record I return?" out of a choice of UTF-8 or MARC-8.
I don't think that's quite right. The encoding param there should be the encoding of the incoming XML. So if the XML is MARC-8 it would unset the UTF-8 flag in the leader and make sure to output MARC-8 bytes (which would be a weird thing to do but that seems to be how it works). But in any case... OP is uploading a .mrc file which will run through C4::Import::RecordsFromISO2709File, which will convert MARC-8 records into UTF-8 records, which OP sort of notes is the encoding Koha uses internally. (In reply to Phil Ringnalda from comment #5)
I'd be happy to sign off a patch that skips setting $encoding and just passes new_from_xml the string "UTF-8", and let QA worry about what on earth someone was thinking passing $self->encoding.
I think you're right about just passing in the string "UTF-8" because at this point the records should be UTF-8 encoded. Honestly looking at C4::ImportBatch::_create_import_record and friends...I have no idea why $encoding is even stored because it's useless at this point. Maybe if it was stored as "source_encoding" but the source file isn't there anyway. Who knows. Looking at 01d78e1ec71 it seems like $self->encoding was used here by accident. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |hammat.wele@inlibro.com |ity.org | CC| |hammat.wele@inlibro.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #168236|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 --- Comment #7 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 170742 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170742&action=edit Bug 37184: Special character encoding problem when importing MARC file from the Acquisitions module 1. Go to Cataloging > "Stage records for import". 2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file. 3. In the form, select the options : - Record type: Bibliographic - Character encoding: MARC8 - Format: MARC 4. Click "Stage for import". 5. Find or create an open basket in the Acquisitions module. 6. Click "add to basket". 7. Select "From a staged file" and select the previous staged file. 9. Click "Select all" to check all records. 10. In the item information tab, choose the Document type. 11. Verify the titles of the records in the basket: --> Koha replaces accents with symbols � 12. Apply the patch. 13. Do the same from step 1 and notice there is no more encoding issues. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170742|0 |1 is obsolete| | --- Comment #8 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 170924 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170924&action=edit Bug 37184: Special character encoding problem when importing MARC file from the Acquisitions module 1. Go to Cataloging > "Stage records for import". 2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file. 3. In the form, select the options : - Record type: Bibliographic - Character encoding: MARC8 - Format: MARC 4. Click "Stage for import". 5. Find or create an open basket in the Acquisitions module. 6. Click "add to basket". 7. Select "From a staged file" and select the previous staged file. 9. Click "Select all" to check all records. 10. In the item information tab, choose the Document type. 11. Verify the titles of the records in the basket: --> Koha replaces accents with symbols � 12. Apply the patch. 13. Do the same from step 1 and notice there is no more encoding issues. Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |normal -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170924|0 |1 is obsolete| | --- Comment #9 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- Created attachment 173527 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173527&action=edit Bug 37184: Special character encoding problem when importing MARC file from the Acquisitions module 1. Go to Cataloging > "Stage records for import". 2. Upload the file "ExportMemento2024061010532869Marc8.mrc" or a MARC8 encoded file. 3. In the form, select the options : - Record type: Bibliographic - Character encoding: MARC8 - Format: MARC 4. Click "Stage for import". 5. Find or create an open basket in the Acquisitions module. 6. Click "add to basket". 7. Select "From a staged file" and select the previous staged file. 9. Click "Select all" to check all records. 10. In the item information tab, choose the Document type. 11. Verify the titles of the records in the basket: --> Koha replaces accents with symbols � 12. Apply the patch. 13. Do the same from step 1 and notice there is no more encoding issues. Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com CC| |martin.renvoize@ptfs-europe | |.com --- Comment #10 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- Looking at the history, I concur.. I would be nice to have a unit test here though -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |Main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Special character encoding |Special character encoding |problem when importing MARC |problem when importing MARC |file from the Acquisitions |file from the acquisitions |module |module -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Martin Renvoize (ashimema) from comment #10)
Looking at the history, I concur..
I would be nice to have a unit test here though
+1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |24.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Pushed to stable CC| |lucas@bywatersolutions.com Version(s)|24.11.00 |24.11.00,24.05.06 released in| | --- Comment #13 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Backported to 24.05.x for upcoming 24.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.11.00,24.05.06 |24.11.00,24.05.06,23.11.11 released in| | Status|Pushed to oldstable |Pushed to oldoldstable CC| |fridolin.somers@biblibre.co | |m --- Comment #14 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37184 Jesse Maseto <jesse@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |jesse@bywatersolutions.com Status|Pushed to oldoldstable |RESOLVED --- Comment #15 from Jesse Maseto <jesse@bywatersolutions.com> --- Not pushed to LTS. Marked Resolved. If you feel this should be in LTS please reply with your reason. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org