https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@libriotech.no -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |RM_priority -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #3 from Jan Kissig <bibliothek@th-wildau.de> --- Hi Michał (In reply to Michał from comment #2)

Are you sure it's a regression and wouldn't happen with the older version of the script?

I just checked it, I ran bulkmarcimport around 20 times before on the same machine, with the same amount of records (~120k) on versions < 24.05. Everytime it worked.

How much swap space does your installation have?

4GB

Generally back when I ran Koha VM with very low RAM (like 2 or 4 GB), some older version like 23.05, I'd have issues with various cron tasks crashing due to OOM (out-of-memory), up until I gave it some significant swap space (like 16 or 24 GB), which I think was even also a recommendation somewhere.

Just thinking if it could perchance be simply a matter of that rather than a regression per se (though it might be both!).

I tried also on ktd with 8GB RAM and it seems that docker is crashing because of bulkmarcimport. I am trying to give it more ressources. I also gave the script a little extra output in both loops and it always stopped in the first while-loop (https://git.koha-community.org/Koha-community/Koha/src/branch/main/misc/migr...) Maybe that @marc_records array is causing my error. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Aditya <adifbbk1@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |adifbbk1@gmail.com --- Comment #5 from Aditya <adifbbk1@gmail.com> --- I am having even bigger problems due to the mysql warnings. I have used koha-dump to backup koha 23.11 in server 1. I have installed latest Koha 24.05 in server 2 and ran koha-restore. As it's a version upgrade, it's expected to have web installer. When I am going through database upgrade step, I am getting errors and can't proceed further: "WARNING: MYSQL_OPT_RECONNECT is deprecated and will be removed in a future version.". I have used "apt install --reinstall koha-common", it seemed like the issue got fixed with the same warning in terminal, but the database doesn't seem to be upgraded as I am now getting "Error 500" in Circulation and Fine Rules page, and empty fields in "Identity Provider" edit buttons, which was working perfectly fine in version 23.11. I have checked through installer codebase, it seems like whenever DBD::mysql generates any warning logs, the Koha's error handling mechanism mistakes it as error instead of warnings (line 441 of "/usr/share/koha/intranet/cgi-bin/installer/install.pl"). And I believe, the same error mechanism is reflected either throughout the Koha codebase or in DBD::mysql package. Possible fix: 1. Change logic for error handling 2. Add "SET sql_notes=0" everywhere before having any query of SQL 3. Bundle error-free DBD::mysql perl package with Koha that has no such issue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com | |, | |kyle@bywatersolutions.com, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Enica Davis <enica@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |enica@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #7 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 176388 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176388&action=edit Bug 37020: Stream records from file instead of loading all into memory One of the issues with bulkmarcimport.pl is that the script loads the entire record set into memory before processing those records. If we load and process those records one at a time the memory needed will be miniscule. Test Plan: 1) Import a very large record set, note the ram consumption be the script 2) Apply this patch 1) Import the same record set, the ram consumption should be reduced! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #176388|0 |1 is obsolete| | --- Comment #8 from Magnus Enger <magnus@libriotech.no> --- Created attachment 176425 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176425&action=edit Bug 37020: Stream records from file instead of loading all into memory One of the issues with bulkmarcimport.pl is that the script loads the entire record set into memory before processing those records. If we load and process those records one at a time the memory needed will be miniscule. Test Plan: 1) Import a very large record set, note the ram consumption be the script 2) Apply this patch 1) Import the same record set, the ram consumption should be reduced! Signed-off-by: Magnus Enger <magnus@libriotech.no> The patch reduces the RAM used when importing a large file. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle.m.hall@gmail.com |ity.org | Status|Signed Off |Failed QA --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Jan, please set to Failed QA (and don't feel bad) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #12 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- (In reply to Katrin Fischer from comment #11)

@Kyle: I think better keep the tidy separate here, hard to see any changes?

There isn't any tidying in this patch ( that I can recall :). It really just involved moving move of the code from the lower loop into the upper loop ( which does change the indentation fwiw ) so there is only one loop instead of two. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #14 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 176653 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176653&action=edit MARCXML with 1 record perl bulkmarcimport.pl -m=MARCXML -b --commit=1000 --file=path/to/bib-262.marcxml -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #16 from Michał <schodkowy.omegi-0r@icloud.com> --- This really highlights the difference of the default patch files as done in Bugzilla. There should be a way to generate a separate git diff that ignores the whitespace changes completely, IDEs can do it, but there doesn't seem to be any popular way to convert an existing diff to skip displaying them... Even displaying the diff with some better viewers like Kompare, it's still not much easier to read. So we can only ask for complimentary git diff -w (--ignore-all-space) attachment to make review easier I guess (I would generate it myself, but I can't clone Koha here on the internet I have today). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #18 from Jan Kissig <bibliothek@th-wildau.de> --- (In reply to Kyle M Hall (khall) from comment #13)

(In reply to Jan Kissig from comment #9)

...

Just tried to import 120k records via

Jan, is there any chance you can attach your records file to this bug?

Hi Kyle, here are the records I tried. Modified to fit ktd. https://nextcloud.th-wildau.de/nextcloud/index.php/s/fB3or6bXNX9fo9E -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |januszop@gmail.com --- Comment #20 from Thomas Klausner <domm@plix.at> --- *** Bug 39537 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #22 from Thomas Klausner <domm@plix.at> --- The patch "Bug 37020: Stream records from file instead of loading all into memory" basically "just" removes the first loop that pushed all (valid) records onto an array. Yes, this solves the memory problem, but still uses the IMO not very sane regex-based pull "parser" in MARC::Batch. But it's a rather small change that works. (even though the patch looks huge because it contains a lot of whitespace-only changes. use `git diff -w ...` to see the relevant changes. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #24 from Thomas Klausner <domm@plix.at> --- In our script, we used XML::LibXML. I can provide an example patch, but before spend that time, here's a quick preview of how the usage would look like: open( my $fh, '<', $input_marc_file ); my $reader = XML::LibXML::Reader->new( IO => $fh ); while ( $reader->read ) { next unless $reader->nodeType == XML_READER_TYPE_ELEMENT; next unless $reader->name eq 'record'; my $xml = $reader->readOuterXml; my $record = MARC::Record->new_from_xml( $xml, 'UTF8', 'MARC21' ); } I find that a bit easier to read than the HTML::Twig way of installing a handler subref for 'record'. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #26 from Thomas Klausner <domm@plix.at> --- One potential killer argument against XML::Twig and in favor of XML::LibXML: Koha currently already comes with XML::LibXML, but not with XML::Twig. So we would need to introduce a new depenceny. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #176425|0 |1 is obsolete| | Attachment #176921|0 |1 is obsolete| | Attachment #180639|0 |1 is obsolete| | --- Comment #28 from David Gustafsson <glasklas@gmail.com> --- Created attachment 180640 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180640&action=edit Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files Include checking for invalid marc data in main loop to avoid loading all marc data into memory at once. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #30 from David Gustafsson <glasklas@gmail.com> --- I think this also resolved a bug where if a limit is set for number of imported records, the loop will terminate prematurely without the last batch of records being committed to Elasticsearch. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #32 from David Gustafsson <glasklas@gmail.com> --- Created attachment 180641 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180641&action=edit Bug 37020: Don't update Elasticsearch index --test option is set -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #180640|0 |1 is obsolete| | Attachment #180641|0 |1 is obsolete| | Attachment #180644|0 |1 is obsolete| | --- Comment #34 from David Gustafsson <glasklas@gmail.com> --- Created attachment 180645 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180645&action=edit Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files Include checking for invalid marc data in main loop to avoid loading all marc data into memory at once. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com Status|Needs Signoff |In Discussion --- Comment #36 from Jonathan Druart <jonathan.druart@gmail.com> --- David, can you reply to Thomas please? See previous comments. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #38 from Thomas Klausner <domm@plix.at> ---

I have had a look at the XML::Twig patch and now that the records are again streamed and not loaded all at once I don't see why we should add a new dependency and increase code complexity with very little to gain in terms of memory consumption.

I agree. Usimg XML::Twig makes little sense, esp as XML::LibXML (and thus XML::LibXML::Reader are already available)

I may be wrong, but memory consumbed by XML::LibXML should be released a record has been read, so there should not really be any significant difference.

The old code (prior to pushing all the records onto an array) had no memory problem. But I would still like to move away from using MARC::Batch when reading XML files, because the code there is crazy. But maybe this could be a second, later patch / bug.

With regard to the patch by Kyle M there is a bug introduced where the main loop can be terminated without indexing the last batch of records, since the conditions for indexing the last batch will not be met. There needs to be an additional check for if we have run out of records, and after that a check whether to terminate the loop.

Yeah, reading that patch was also quite hard... Anyway, I think what we should do is: * review your patch * merge it if it works * start another bugzilla issue to refactor bulkmarcimport to use XML::LibXML::Reader for streaming XML input files -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #40 from Leo Stoyanov <leo.stoyanov@bywatersolutions.com> --- Sounds like a good plan to me as well: If Dave's patch solves the immediate issue, then that could be the official patch for this bug; then, another Bugzilla issue could be made for implementing XML::LibXML::Reader as an enhancement. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #42 from Thomas Klausner <domm@plix.at> --- Created attachment 180905 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180905&action=edit Bug 37020: (Followup) Fix txn handling, loop label and record_number counter So, I reviewed the code, which has a bunch of issues: * there's a call to `next RECORD`, but no `RECORD` loop label. * $record_number is incremented twice * and the calls to txn_begin is moved from the outside the loop into each record, starting one transaction per record. Not sure how mysql handles this, but postgres would not like it. This patch fixes those issues: After I fixed the first two issues, I ran the patched script on a file containing 75 authorities (from GND), and not one was added to the DB, presuambly because nested transactions don't work in mysql? So I also moved the BEGIN transaction outside of the main loop and started a new one after each commit, and added a final commit after to loop. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #44 from Thomas Klausner <domm@plix.at> --- Here'a test plan: 1: Start KTD with ES, eg ktd --es7 start 2: Connect to DB: ktd --dbshell 3: Get the count of current auth: `SELECT count(*) FROM auth_header;` I get 1706. Remember that number 4: Apply the patch, copy the file "MARCXML file with 75 authorities from GND" / 75_authorities_gnd.xml into your koha dir 5: enter a koha shell: ktd --shell 6: import the 75_authorities_gnd.xml file: kohadev-koha@kohadevbox:koha$ misc/migration_tools/bulkmarcimport.pl -m MARCXML -c utf8 -a --insert --file 75_authorities_gnd.xml -l testlog You will see two DBIx errors, but at the end: 75 MARC records done in 0.464853048324585 seconds 7: Again in the DB-Shell, run the count: SELECT count(*) FROM auth_header; You should now get 75 more auths (eg 1781 in my ktd) you can verify that in the DB with: select 1781 - 1706; 8: Take a look at the testlog file, which will have 73 OKs and two errors : eg: 1713;insert;ok 000152145;insert;ERROR 1715;insert;ok Notice that 1714 is skipped (because it had an error). But it still exists in the DB (in a useless form): select * from auth_header where authid = 1714; I'm not exactly sure why the two not importable auths show up in the db. But as this is the same behavior as on main (without this patch) I consider it out of scope for this bug. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #46 from Thomas Klausner <domm@plix.at> ---

is it ready to get tested with a large number of (bib)records

Yes! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #48 from Jan Kissig <bibliothek@th-wildau.de> --- Today closed all applications except the running ktd and tried the file from https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020#c18 again. My setup 2017 I7 with 16GB RAM. Although it lasts longer than yesterday it got killed anyway: perl misc/migration_tools/bulkmarcimport.pl -d -b -commit=100 -m=MARCXML -file=../thw/btw.xml ... 120400.................................................................................................... 120500.................................................................................................... 120600...................................................Killed -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #50 from Thomas Klausner <domm@plix.at> --- Here is a authorities MARC file with 42k records: https://data.dnb.de/GNDlfdMarc21xml/2515gndmrc.xml.gz Here's the command I use to import it into koha-testing-docker (via `ktd --shell` and fresh db): misc/migration_tools/bulkmarcimport.pl -m MARCXML -v -a --file 2515gndmrc.xml After starting it, get the pid of the process, eg via `ps xa | grep bulkmarc` (in the container or on the node, does not matter) The watch the memory usage, eg via `htop -p $PID` or while ps -p $PID --no-headers --format "etime pid %cpu %mem rss"; do sleep 1 ; done I see hardly any growth in RAM usage (htop reports 217M RES; the `while ps` bash thingy reports 222340 (which is basically the same) And here's the result without the patches applied: RAM usages goes up to 1620MB (so 1.6GB) or 1648108, which is what it takes to store the data in the array (and which is fixed by this patch) and then continues to slowly grow. So: a) the patches definitely work (at least for auths...) and b) there are other mem leaks. I will now try to download your big file btw.xml and see how it works on my laptop. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #52 from Thomas Klausner <domm@plix.at> --- Created attachment 181463 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181463&action=edit Bug 37020: (follow-up): Remove memleak when not using Elasticsearch If using ES, the script will collect the newly created records and biblionumbers in atwo arrays (@search_engine_record, @search_engine_record_ids) and pass that to the indexer when $commitnum records are read. The arrays are cleaned afterwards. BUT: The records and biblionumbers were always stored, even if we're not using Elasticsearch. And as the cleaning of those arrays only happens if we are in fact using ES, all the data is added but never cleaned. This patch fixes this by only storing record and biblionumber if we have an $indexer (i.e. if we're using Elasticsearch). I have no idea though how/if the records are indexed when using Zebra. After applying this patch and importing a large enough file without using Elasticsearch, the RAM usage also stays more or less constant. @Jan, please verify! Sponsored-by: HKS3 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #180645|0 |1 is obsolete| | Attachment #180905|0 |1 is obsolete| | Attachment #181463|0 |1 is obsolete| | --- Comment #53 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 181470 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181470&action=edit Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files Include checking for invalid marc data in main loop to avoid loading all marc data into memory at once. Signed-off-by: Jan Kissig <bibliothek@th-wildau.de> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #55 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 181472 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181472&action=edit Bug 37020: (follow-up): Remove memleak when not using Elasticsearch If using ES, the script will collect the newly created records and biblionumbers in atwo arrays (@search_engine_record, @search_engine_record_ids) and pass that to the indexer when $commitnum records are read. The arrays are cleaned afterwards. BUT: The records and biblionumbers were always stored, even if we're not using Elasticsearch. And as the cleaning of those arrays only happens if we are in fact using ES, all the data is added but never cleaned. This patch fixes this by only storing record and biblionumber if we have an $indexer (i.e. if we're using Elasticsearch). I have no idea though how/if the records are indexed when using Zebra. After applying this patch and importing a large enough file without using Elasticsearch, the RAM usage also stays more or less constant. @Jan, please verify! Signed-off-by: Jan Kissig <bibliothek@th-wildau.de> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #181470|0 |1 is obsolete| | --- Comment #56 from Thomas Klausner <domm@plix.at> --- Created attachment 181996 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181996&action=edit Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files Include checking for invalid marc data in main loop to avoid loading all marc data into memory at once. Signed-off-by: Jan Kissig <bibliothek@th-wildau.de> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #181472|0 |1 is obsolete| | --- Comment #58 from Thomas Klausner <domm@plix.at> --- Created attachment 181998 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181998&action=edit Bug 37020: (follow-up): Remove memleak when not using Elasticsearch If using ES, the script will collect the newly created records and biblionumbers in atwo arrays (@search_engine_record, @search_engine_record_ids) and pass that to the indexer when $commitnum records are read. The arrays are cleaned afterwards. BUT: The records and biblionumbers were always stored, even if we're not using Elasticsearch. And as the cleaning of those arrays only happens if we are in fact using ES, all the data is added but never cleaned. This patch fixes this by only storing record and biblionumber if we have an $indexer (i.e. if we're using Elasticsearch). I have no idea though how/if the records are indexed when using Zebra. After applying this patch and importing a large enough file without using Elasticsearch, the RAM usage also stays more or less constant. @Jan, please verify! Signed-off-by: Jan Kissig <bibliothek@th-wildau.de> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Patch doesn't apply --- Comment #60 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Please rebase on current main: Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files error: sha1 information is lacking or useless (misc/migration_tools/bulkmarcimport.pl). error: could not build fake ancestor Patch failed at 0001 Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-37020-bulkmarcimport-gets-killed-after-update--zzl30sda.patch -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|kyle.m.hall@gmail.com |domm@plix.at -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Passed QA --- Comment #61 from Thomas Klausner <domm@plix.at> --- One patch was marked as obsolete, but it wasn't (this was probably my fault during some review/signing step) I have now un-obsoloted the patch Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files (33.43 KB, patch) 2025-04-04 13:28 UTC, David Gustafsson And now all 4 patches apply again without a conflict! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #63 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files Applying: Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files error: sha1 information is lacking or useless (misc/migration_tools/bulkmarcimport.pl). error: could not build fake ancestor Patch failed at 0001 Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-37020-bulkmarcimport-gets-killed-after-update--m_neycks.patch -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #180645|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|major |critical -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #66 from Thomas Klausner <domm@plix.at> --- I have now pushed the three commits / patches to: https://github.com/domm/Koha/tree/37020_bulkmarcimport @cait: I hope you can work with this version? Or should I open a pull request on the Koha repo on github? (Meta: I haven't changed the status of the issue here in bugzilla) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #67 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Patches applied without issue from remote branch now. But noting: Here we have 4 patches, on the branch there were 3. Squashed? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Passed QA --- Comment #69 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Thomas Klausner from comment #68)

I think two of my fixed got somehow merged into one commit. So yes, all OK (esp as the actual main patch by David Gustafsson is properly credited)

I just noticed that this had the wrong status :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Paul Derscheid <paul.derscheid@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|25.05.00 |25.05.00,24.11.05 released in| | Status|Pushed to main |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Paul Derscheid <paul.derscheid@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.derscheid@lmscloud.de Keywords|rel_24_11_candidate | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 --- Comment #73 from Jonathan Druart <jonathan.druart@gmail.com> --- record_number is still incremented twice, is this because of a conflict fix problem? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Alexander Wagner <alexander.wagner@desy.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alexander.wagner@desy.de --- Comment #75 from Alexander Wagner <alexander.wagner@desy.de> --- If I get the code right, the number of records ingested is still doubled. AFAIS `$record_number++;` is called twice within the loop. Jan mentioned that he ingested 1 record and got notified about 3, I also obeserved a result of 5 while I had only two (valid) records in the import file. Might be a minor issue compared to all the rest but can be quite confusing if your catalogue migration turns out to hold way more records that you expected. (Koha 25.05.00 ga) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #76 from Fridolin Somers <fridolin.somers@biblibre.com> --- Still additional_work_needed ? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pedro.amorim@openfifth.co.u | |k Keywords|additional_work_needed | --- Comment #78 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Alexander, Joubu, if there is a bug here please file a separate report, removing additional_work_needed. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41681 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 [Bug 41681] bulkmarcimport.pl reports an incorrect number of MARC records processed -- You are receiving this mail because: You are watching all bug changes.