https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major --- Comment #2 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Upping severity. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 96160 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96160&action=edit Bug 24123: Fix import of UTF-8 encoded MARC21 MARCXML using bulkmarcimport (elastic only) If elastic is used as search engine, the bulkmarcimport.pl will not handle correctly UTF-8 encoded MARCXML Koha::SearchEngine::Search->new uses a require statement to load the correct Search module. This is done l.257 of bulkmarcimport.pl: 257 my $searcher = Koha::SearchEngine::Search->new Koha::SearchEngine::Elasticsearch::Search will `use MARC::File::XML`, and so resets the arguments set before: 216 $MARC::File::XML::_load_args{BinaryEncoding} = 'utf-8'; 220 $MARC::File::XML::_load_args{RecordFormat} = $recordformat; An easy (but dirty) fix could be to move the declaration of my $searcher before in the script. The tricky (but correct) fix would be to remove the long standing "ugly hack follows" comment. This patch is the easy, and dirty, fix Test plan: Use the command line tool to import MARXCML records that contains unicode characters into Koha Something like `misc/migration_tools/bulkmarcimport.pl -biblios -file record.marcxml -m=MARCXML` Without this patch you will notice that unicode characters will not be displayed correctly -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |black23@gmail.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #96160|0 |1 is obsolete| | --- Comment #4 from Michal Denar <black23@gmail.com> --- Created attachment 96967 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96967&action=edit Bug 24123: Fix import of UTF-8 encoded MARC21 MARCXML using bulkmarcimport (elastic only) If elastic is used as search engine, the bulkmarcimport.pl will not handle correctly UTF-8 encoded MARCXML Koha::SearchEngine::Search->new uses a require statement to load the correct Search module. This is done l.257 of bulkmarcimport.pl: 257 my $searcher = Koha::SearchEngine::Search->new Koha::SearchEngine::Elasticsearch::Search will `use MARC::File::XML`, and so resets the arguments set before: 216 $MARC::File::XML::_load_args{BinaryEncoding} = 'utf-8'; 220 $MARC::File::XML::_load_args{RecordFormat} = $recordformat; An easy (but dirty) fix could be to move the declaration of my $searcher before in the script. The tricky (but correct) fix would be to remove the long standing "ugly hack follows" comment. This patch is the easy, and dirty, fix Test plan: Use the command line tool to import MARXCML records that contains unicode characters into Koha Something like `misc/migration_tools/bulkmarcimport.pl -biblios -file record.marcxml -m=MARCXML` Without this patch you will notice that unicode characters will not be displayed correctly Signed-off-by: Michal Denar <black23@gmail.com> Nice work, Jonathan -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #96967|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 97138 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=97138&action=edit Bug 24123: Fix import of UTF-8 encoded MARC21 MARCXML using bulkmarcimport (elastic only) If elastic is used as search engine, the bulkmarcimport.pl will not handle correctly UTF-8 encoded MARCXML Koha::SearchEngine::Search->new uses a require statement to load the correct Search module. This is done l.257 of bulkmarcimport.pl: 257 my $searcher = Koha::SearchEngine::Search->new Koha::SearchEngine::Elasticsearch::Search will `use MARC::File::XML`, and so resets the arguments set before: 216 $MARC::File::XML::_load_args{BinaryEncoding} = 'utf-8'; 220 $MARC::File::XML::_load_args{RecordFormat} = $recordformat; An easy (but dirty) fix could be to move the declaration of my $searcher before in the script. The tricky (but correct) fix would be to remove the long standing "ugly hack follows" comment. This patch is the easy, and dirty, fix Test plan: Use the command line tool to import MARXCML records that contains unicode characters into Koha Something like `misc/migration_tools/bulkmarcimport.pl -biblios -file record.marcxml -m=MARCXML` Without this patch you will notice that unicode characters will not be displayed correctly Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |20.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Martin Renvoize from comment #6)

Happy with this pragmatic approach to a fix..

It has however, bugged me for a long time that we have both bulkmarcimport and the stage + commit command-line scripts to do the same process in reality. I'd love to see that tidied up some time.

Also.. I'm really not sure why we need MARC::Batch at all.. my understanding is that its intended use case is for multiple files, but we're only loading a single file using bulkmarcimport aren't we?

Either way, pushing :)

While both import, they do it quite differently. Bulkmarcimport is faster as it does not use the staging tables, but imports directly into the database. Matching works differently (not using the matching rules system) and it can't be undone. The stage and commit scripts could probably be combined into one thing, but as they are they match the fact it's also 2 tools in Koha. They use the matching rules and import tables, which can mean create quite a lot of data db size wise as everything is stored twice after importing and cleaning the batch I think doesn't really make the db shrink. But you can undo them and use complex matching rules. When you want to have 'versioning' for your records, it's also helpful to keep that data. We noticed that loading big files via stage/commit created quite a big load on the servers - I wonder if instead of multiple files a possible use could be to break up big files and work on them sequentially? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24123 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|20.05.00 |20.05.00, 19.11.03 released in| | CC| |joy@bywatersolutions.com Status|Pushed to master |Pushed to stable --- Comment #10 from Joy Nelson <joy@bywatersolutions.com> --- Pushed to 19.11.x branch for 19.11.03 -- You are receiving this mail because: You are watching all bug changes.