[Bug 17791] New: On UNIMARC system, USMARC field 100 gets overwritten
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Bug ID: 17791 Summary: On UNIMARC system, USMARC field 100 gets overwritten Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Z39.50 / SRU / OpenSearch Servers Assignee: gmcharlt@gmail.com Reporter: synapse.ova@gmail.com QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl Overview: On Z39.50 search some functions depend not on the remote server marc flavour, but on current server setting. This bug is present on versions 3.22+ Steps to Reproduce: 1) On UNIMARC system, go to Cataloguing → New from Z39.50/SRU 2) Search USMARC server (Library of Congress, for example) for a book, that has field 100 (for example, isbn 0816310653) 3) On results list, press «MARC» to see MARC fields Actual Results: Instead of seeing the author name in subfield A, you'll see something like «20161219 frey50 », yet you could see other subfields properly Expected Results: Name of the author in 100$a Additional Information: LC Z39.50/SRW/SRU Server Configuration Guidelines: https://www.loc.gov/z3950/lcserver.html -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Oleg Vasilenko <synapse.ova@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|On UNIMARC system, USMARC |On UNIMARC system, USMARC |field 100 gets overwritten |field 100$a gets | |overwritten -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 --- Comment #1 from Oleg Vasilenko <synapse.ova@gmail.com> --- Created attachment 58276 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58276&action=edit Patch changes the context of Z39.50 search from local to remote server marc flavour -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Oleg Vasilenko <synapse.ova@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|--- |Sponsored Status|NEW |Needs Signoff Patch complexity|--- |String patch --- Comment #2 from Oleg Vasilenko <synapse.ova@gmail.com> --- Patch prepared for current master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.tajoli@cineca.it Status|Needs Signoff |Failed QA --- Comment #3 from Zeno Tajoli <z.tajoli@cineca.it> --- Hi Oleg, I'm not agree with this patch. Why? Now «20161219 frey50 » is correct default for Unimarc 100 field If you have a Unimarc server and you want to use LOC [a MARC21 z39.50] you need to use the field "XSLT file(s)" in z39.50 definiton. See: http://manual.koha-community.org/3.22/en/additionaladmin.html#addztarget In bug 6536 there is an example of XSLT for MARC21 -> Unimarc conversion. See: https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37264 Bye -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Oleg Vasilenko <synapse.ova@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #4 from Oleg Vasilenko <synapse.ova@gmail.com> --- Hi Zeno, This change is normal if you access UNIMARC record on Unimarc server. But the problem is that this is changed before XSLT conversion. I do xslt conversion, but it converts this already changed string. You can check by yourself https://github.com/Gelow/unimarc/blob/master/marc21xml2unimarc.xsl MARC view in Z39.50 search results shows the record as it is. Koha code works the way, that it first converts codepage, and after that changes the subfield where it is stated, that it is UTF-8. But the problem is that it thinks that the record is in unimarc format, so it changes 100$a. The patch uses server flavour, that is stated in Z39.50 servers table. That way everythig works fine. Function, that do codepage conversion understands that this record is USMARC (as stated in Z39.50 server table), and it changes leader, not 100$a. Because information about codepage is located in different places in these flavours. You can see that this problem is present in different koha versions. But the patch is prepared for current master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 --- Comment #5 from Zeno Tajoli <z.tajoli@cineca.it> --- Ok Oleg, you are correct. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 --- Comment #6 from Oleg Vasilenko <synapse.ova@gmail.com> --- When you use XSLT conversion, you'll see the same problem USMARC 100 is converted to UNIMARC 700. But the conversion happens after codepage is changed. So you'll see 700$a 20161219 frey50 700$b 1933- but not the author name in 700$a -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 --- Comment #7 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Oleg, This seems to me an tricky change. What you are trying to do, is currently not fully supported in Koha. A conversion between MARC21 and UNIMARC did not make it until now. If you do a XSLT conversion after this statement, it will probably be fine, but in many other cases it will not be followed by such a conversion and new problems will probably arise. To make this foolproof, we need more changes (and checks)? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 --- Comment #8 from Oleg Vasilenko <synapse.ova@gmail.com> --- Marcel, I am working on the MARC21 to UNIMARC conversion right now, and now it works good enough. I did many changes to original XSLT file published by Terry Reese on github (bug 16488). See https://github.com/Gelow/unimarc/blob/master/marc21xml2unimarc.xsl The only problem is that field 100$a is corrupted on Z39.50 search. I added LOC server as SRU and field 100$a is not corrupted that way. This is because in sub _handle_one_result() right now SRU server is processed with it's flavour in mind. The same should be with Z39.50 What problems will we face after this patch? The same as we have with SRU servers right now. When UNIMARC system connects to MARC21 server, it will check encoding stated in 100$a (thanks to checks in XML.pm called from function AddBiblioToBatch()).If it's not set, you'll see XML encoding error («Unsupported UNIMARC character encoding [$enc] for XML output»). In other cases everything should be fine If we do not apply this patch You can see the results table, and see sourse MARC with corrupted 100$a. But the table has empty columns because of different format, so who does need it? To overcome stated problems it is possible to add some check in the code for UNIMARC and process the record some way (but record could become corrupted again). Or add XSLT file for conversion -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |chris@bigballofwax.co.nz Severity|major |new feature --- Comment #9 from Chris Cormack <chris@bigballofwax.co.nz> --- Changing to a new feature as this is adding support for on the fly conversion of MARC records from Z39.50 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Serhij Dubyk <dubyk@ukr.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dubyk@ukr.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de Severity|new feature |enhancement --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Changing to enh as this is just the starting point for adding the ability of the conversion (and should then change bug title also). After this made it in we could maybe add Oleg's XSLT file to the sample xsl files available and mark that 'new feature'? Signing off, was not able to detect a regression. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 68165 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68165&action=edit Bug 17791 - On UNIMARC system, USMARC field 100$a gets overwritten Patch changes the context of Z39.50 search from local to remote server Steps to Reproduce: 1) On UNIMARC system, go to Cataloguing → New from Z39.50/SRU 2) Search USMARC server (Library of Congress, for example) for a book, that has field 100 (for example, isbn 0816310653) 3) On results list, press «MARC» to see MARC fields Expected Results: Name of the author in 100$a Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|On UNIMARC system, USMARC |On UNIMARC system, USMARC |field 100$a gets |field 100$a gets |overwritten |overwritten when importing | |from MARC21 target -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58276|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |synapse.ova@gmail.com QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | Status|Signed Off |Failed QA --- Comment #12 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- This does not work. As said earlier, we need more checks. (The records cannot be imported without conversion. So we need more checks. For instance, only show import when the flavours match.) Currently btw this results in an internal server error. I started on a clean master, chose UNIMARC and added LOC as a search target (MARC21/USMARC). Without this patch, I can search, but of course I do not see any titles. With this patch, it crashes. Unrecognized marcflavour: USMARC at /usr/share/koha/masterclone/C4/Charset.pm line 358. Use of uninitialized value in substr at /usr/share/perl5/MARC/File/XML.pm line 561. substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 561. Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 563. Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 563. Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565. Use of uninitialized value $enc in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 568. Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 568. Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> at /usr/share/perl5/MARC/File/XML.pm line 568. Script passes USMARC as marc flavour to MarcToUTF8Record. This routine passes it to SetMarcUnicodeFlag which expects MARC21 or UNIMARC as Koha does in other places too. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |master Patch complexity|String patch |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathsabypro@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #68165|0 |1 is obsolete| | --- Comment #13 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 176866 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176866&action=edit Bug 17791: Fix Z39.50/XSLT import encoding issue Test plan: 1. On UNIMARC system, create a Z39.50 server configuration: Name: LOC_NO_XSLT Hostname: lx2.loc.gov Port: 210 Database: LCDB Preselected: yes Syntax: MARC21/USMARC Encoding: utf8 Record Type: bibliographic 2. Duplicate this Z39.50 server configuration, change its name to "LOC_WITH_XSLT" and set "XSLT File(s) for transforming results" to the absolute path of a MARC21 to UNIMARC XSL file. You can download one from here: https://raw.githubusercontent.com/Gelow/unimarc/refs/heads/master/marc21xml2... 3. Go to Cataloguing → New from Z39.50/SRU 4. Search both servers for a book entitled "comprendre un pays". That should give you only one result for each server 5. On results list, press «MARC» to see MARC fields. The record from LOC_NO_XSLT should have its 100$a replaced by something like "20250121 frey50" and the rest of the record should be a valid MARC21 record The record from LOC_WITH_XSLT should also have a 100$a looking like "20250121 frey50" but it might be different depending on the XSLT used (the important part is the "50" at positions 26-27) It should also have a 700 field that look like this: $a Simon $b Jean-François but again, it depends on the XSLT used. The 700$a should NOT contain "20250121 frey50" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com Status|Failed QA |Needs Signoff Assignee|synapse.ova@gmail.com |julian.maurice@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 --- Comment #14 from Mathieu Saby <mathsabypro@gmail.com> --- I tried in k-t-d, the results seem to vary, I don't understand why (instability of the connection to the Z3950 server?), and the version “with xslt” generates an error (I used the xslt sheet given in the test plan). Without XSLT, one time I get a record with only a leader a 100$a field LDR 00079 a2200037 4500 100 _a20250209 frey50 A few minutes later I get a MARC21 record : 100 1 _a20250209 frey50 _d1950- _eauthor. 245 10 _aComprendre un pays : _bethnologie de la Basse-Bretagne / _cJean-François Simon. With XSLT, one time I have that error Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> at /usr/share/perl5/MARC/File/XML.pm line 570 in MARC::File::XML::_unimarc_encoding at /usr/share/perl5/MARC/File/XML.pm line 570 567: } elsif ($enc eq '50') { 568: return 'UTF-8'; 569: } else { 570: die "Unsupported UNIMARC character encoding [$enc] for XML output for $f; 100\$a -> " . $r->subfield(100 => 'a'); 571: } 572: } 573: Show function arguments An other time that error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> Simon, Jean-Fran�ois, at /usr/share/perl5/MARC/File/XML.pm line 570 in MARC::File::XML::_unimarc_encoding at /usr/share/perl5/MARC/File/XML.pm line 570 567: } elsif ($enc eq '50') { 568: return 'UTF-8'; 569: } else { 570: die "Unsupported UNIMARC character encoding [$enc] for XML output for $f; 100\$a -> " . $r->subfield(100 => 'a'); 571: } 572: } 573: Show function arguments -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17791 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org