[Bug 38908] New: UNIMARC Koha install does not work for Elasticsearch
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 Bug ID: 38908 Summary: UNIMARC Koha install does not work for Elasticsearch Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching Assignee: koha-bugs@lists.koha-community.org Reporter: mathsabypro@gmail.com QA Contact: testopia@bugs.koha-community.org Trying to launch a UNIMARC Koha with ktd with "--es7" and "--es8" options, I have this error. Not sure if it is a ktd bug or a Koha bug... Can't locate object method "type" via package "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50 ba0 at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | [1219] Indexing biblios koha-1 | [1219] 1000 records processed koha-1 | [1219] 2000 records processed koha-1 | [1219] 3000 records processed koha-1 | [1219] 4000 records processed koha-1 | [1219] Committing final records... es-1 | {"type": "server", "timestamp": "2025-01-16T14:54:41,756Z", "level": "INFO", "component": "o.e.c.m.MetadataMappingService", "cluster.name": "docker-cluster", "node.name": "37065e538e28", "message": "[koha_kohadev_biblios/1QeJV2h3R-SPTVkcuxG8TQ] update_mapping [_doc]", "cluster.uuid": "0PUajrC7QTuY5ta--TrqVw", "node.id": "Q2nG1GyeSqmrNhMk5n4nrQ" } koha-1 | [1219] Total 4849 records indexed koha-1 | [1219] Indexing authorities koha-1 | [1219] 1000 records processed koha-1 | [1219] 2000 records processed koha-1 | [1219] 3000 records processed koha-1 | [1219] 4000 records processed koha-1 | [1219] 5000 records processed koha-1 | [1219] Committing 5000 records... koha-1 | Warnings encountered while roundtripping a MARC record to/from USMARC. Failing over to MARCXML. at /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl line 317. koha-1 | substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 563. koha-1 | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565. koha-1 | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565. koha-1 | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 567. koha-1 | Use of uninitialized value $enc in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | Can't locate object method "type" via package "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50 ba0 at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | " (perhaps you forgot to load "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50 ba0 at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | "?) at /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl line 321. db-1 | 2025-01-16 14:54:46 12 [Warning] Aborted connection 12 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets) db-1 | 2025-01-16 14:54:46 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets) db-1 | 2025-01-16 14:54:46 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #1 from David Nind <david@davidnind.com> --- I can confirm that I get the same result with ES7 and ES8. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #2 from David Nind <david@davidnind.com> --- I also tried checking out Koha 24.11.00 (git checkout tags/v24.11.00), instead of just main, and got the same error as well. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|UNIMARC Koha install does |KTD - UNIMARC Koha start up |not work for Elasticsearch |does not work for | |Elasticsearch --- Comment #3 from David Nind <david@davidnind.com> --- I also got the same error with both ES7 and ES8 with 24.05.x. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #4 from Mathieu Saby <mathsabypro@gmail.com> --- Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #5 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Mathieu Saby from comment #4)
Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ?
I am wondering if this could relate to bug 39013. Could you try leaving off es and start with Zebra? Does this work? If it's a ktd issue, we need to move the bug to the ktd issue tracker later. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #6 from Mathieu Saby <mathsabypro@gmail.com> --- It works well with Zebra Maybe linked with https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38416 ? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Katrin Fischer from comment #5)
(In reply to Mathieu Saby from comment #4)
Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ?
I am wondering if this could relate to bug 39013. Could you try leaving off es and start with Zebra? Does this work?
If it's a ktd issue, we need to move the bug to the ktd issue tracker later.
Sorry typo in bug number: bug 38913 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #8 from Mathieu Saby <mathsabypro@gmail.com> --- I tried to apply bug 38913 with no effect But in a Zebra Unimarc KTD I run a report and found 10 records where the 100$a field seems to have 35 bytes and not 36. Biblionumbers 1651, 336, 3116, 4001, 3950, 3685, 4141, 3012, 3700, 212 Could this be the cause of the issue? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |jonathan.druart@gmail.com --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Mathieu Saby from comment #8)
I tried to apply bug 38913 with no effect
But in a Zebra Unimarc KTD I run a report and found 10 records where the 100$a field seems to have 35 bytes and not 36. Biblionumbers 1651, 336, 3116, 4001, 3950, 3685, 4141, 3012, 3700, 212
Could this be the cause of the issue?
I wonder why it suddenly fails now, but having the wrong length seems a data issue. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #10 from Mathieu Saby <mathsabypro@gmail.com> --- I open this issue in gitlab : https://gitlab.com/koha-community/koha-misc4dev/-/issues/84 Does the current bug need to be closed, or maybe there is patch to add to avoid an indexing failure in the case of bad length in that field ? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |MOVED URL| |https://gitlab.com/koha-com | |munity/koha-misc4dev/-/issu | |es/84 Status|NEW |RESOLVED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |a.roussos@dataly.gr Resolution|MOVED |--- Status|RESOLVED |REOPENED --- Comment #11 from Andreas Roussos <a.roussos@dataly.gr> --- Following the work done in https://gitlab.com/koha-community/koha-misc4dev/-/merge_requests/74 by Mathieu Saby to fix subfield 100$a for certain biblionumbers, we were still unable to launch a UNIMARC-based Koha instance using the latest koha-testing-docker container image (the kohadev-koha-1 container would exit during the authorities ES indexing process). Eventually, I managed to make it work by checking out a branch on my local kohaclone at a specific commit (1c3ed6e4db) prior to running the `ktd --es7 up` command. That particular commit was chosen because it's immediately before commit 0d862343dd (Bug 38416: Failover to MARCXML if cannot roundtrip USMARC during indexing). What lead us to this commit/bug report was the "Warnings encountered while roundtripping a MARC record to/from USMARC" line in the container logs at the end of the authorities ES indexing process. Obviously this is just a workaround and not a real fix, but at least we can now launch `ktd --shell` to access the Koha container and investigate further. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #12 from Andreas Roussos <a.roussos@dataly.gr> --- Apparently, the majority of UNIMARC sample authority records from koha-misc4dev have an extra space at the end of the leader value (in both their MARCXML and the MARC/ISO2709 representation). If you remove the extra space (making the leader length equal to 24 characters), the ES indexing completes without any warnings/errors: $ ktd --shell k$ cd /kohadevbox/misc4dev/data/sql/unimarc/1812/after_22155 k$ sed -i 's/ 4500 <\/leader>/ 4500<\/leader>/g' auth_header.sql k$ sed -E -i 's/([0-9]{5})([cdn])([xyz]) ([a-m])22([0-9]{5}) 4500 /\1\2\3 \422\5 4500/g' auth_header.sql k$ reset_all [...] Running [sudo koha-shell kohadev -p -c 'PERL5LIB=/kohadevbox/koha:/kohadevbox/koha/lib:/kohadevbox/qa-test-tools perl /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl -v']... [343545] Checking state of biblios index [343545] Checking state of authorities index [343545] Indexing biblios [343545] 1000 records processed [343545] 2000 records processed [343545] 3000 records processed [343545] 4000 records processed [343545] Committing final records... [343545] Total 4849 records indexed [343545] Indexing authorities [343545] 1000 records processed [343545] 2000 records processed [343545] 3000 records processed [343545] 4000 records processed [343545] 5000 records processed [343545] Committing 5000 records... [343545] Commit complete [343545] 6000 records processed [343545] Committing final records... [343545] Total 6635 records indexed [...] -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #13 from Jonathan Druart <jonathan.druart@gmail.com> --- Andreas, could you provide a merge request for https://gitlab.com/koha-community/koha-misc4dev ? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 --- Comment #14 from Andreas Roussos <a.roussos@dataly.gr> --- (In reply to Jonathan Druart from comment #13)
Andreas, could you provide a merge request for https://gitlab.com/koha-community/koha-misc4dev ? I sure can, Jonathan!
Please see: https://gitlab.com/koha-community/koha-misc4dev/-/merge_requests/75 This was my first ever merge request on GitLab, I hope I got everything right... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38908 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |MOVED --- Comment #15 from Andreas Roussos <a.roussos@dataly.gr> --- UNIMARC KTD instances are now working again ;-) Tested with `ktd --es7 up` and `ktd --es8 up` after a `ktd pull`. Marking this Bug as 'RESOLVED'. Thanks to all! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org