https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 AspenCat Team <aspencatteam@clicweb.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aspencatteam@clicweb.org --- Comment #3 from AspenCat Team <aspencatteam@clicweb.org> --- We would support this move as well, it would result in fewer tickets to ByWater saving them time, and increase the turnaround time for our libraries to get data loaded. I am sure there are other library users and support companies in this situation who could also benefit. Ultimately this would be a cost savings investment for anyone wanting to take this development on. - Bob Bennhoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |pedro.amorim@openfifth.co.u |ity.org |k CC| |pedro.amorim@openfifth.co.u | |k -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #7 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200599 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200599&action=edit Bug 34632: Add Koha::BackgroundJob::PatronImport Introduces a new background job class for patron imports. enqueue() reads the uploaded file content into the job_args and uses line count as job_size for progress tracking. process() reconstitutes the file handle from the stored content, delegates to Koha::Patrons::Import with step_callback wired to $self->step, handles patron list creation, stores the import results, and clears file_content from the stored data to free space. Also registers patron_import in Koha::BackgroundJob::core_types_to_classes so _derived_class can resolve the job correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #9 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200601 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200601&action=edit Bug 34632: Use PatronImport background job in import_borrowers import_borrowers.pl now reads the uploaded file into memory and calls Koha::BackgroundJob::PatronImport->enqueue() instead of running the import synchronously. On success the template shows a confirmation with a link to the background job viewer; on failure it shows an error alert. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #11 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200603 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200603&action=edit Bug 34632: [DO NOT PUSH] Patron import sample data To be used to upload at <staff_url>/cgi-bin/koha/tools/import_borrowers.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Brendan Lawlor <blawlor@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off Sponsorship status|--- |Sponsored Comma delimited| |CLAMS list of Sponsors| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #12 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200660 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200660&action=edit Bug 34632: Add step_callback support to Koha::Patrons::Import Adds an optional step_callback coderef parameter to import_patrons(). When provided it is called after every row — whether the row succeeds, is skipped as a duplicate, or is rejected as invalid — so callers can track progress. Also removes the 25-error cap that was there to protect browser response size; that constraint no longer applies when results are stored in a background job's data field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #14 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200662 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200662&action=edit Bug 34632: Add tests for Koha::BackgroundJob::PatronImport Tests cover enqueue() (status, queue, job_size, job_type) and six process() scenarios: happy path, import errors (with validation that the 25-error cap has been removed), already_in_db and overwritten, patron list creation, and cancelled job handling. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #16 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200664 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200664&action=edit Bug 34632: Add patron_import background job display template patron_import.inc provides the report/detail/js blocks consumed by the background job viewer: a summary of counts when finished, a live progress bar while running, and the full feedback and error analysis in the detail section. background_jobs.tt gains a patron_import entry in the job type filter dropdown. Test plan: 1) Apply patches, restart plack. 2) Navigate to the patron import tool: <staff_url>/cgi-bin/koha/tools/import_borrowers.pl 3) Upload the provided sample CSV file and click 'Import patrons'. 4) Confirm the page shows "The patron import job has been enqueued!" with a link to view the job. 5) Follow the link to the background job. Confirm a progress bar is shown while the job is running. 6) Once finished, confirm the report shows the correct counts (imported, overwritten, already_in_db, invalid, total). 7) Confirm the imported patrons exist under 'Patrons'. 8) Re-import the same file with 'Overwrite cardnumber' unchecked. Confirm 'already_in_db' count is 5. 9) Re-import again with 'Overwrite cardnumber' checked. Confirm 'overwritten' count is 5. 10) Visit the background jobs admin and filter by type 'Import patrons': <staff_url>/cgi-bin/koha/admin/background_jobs.pl 11) Confirm the jobs appear in the list. 12) Run the tests: prove t/db_dependent/Koha/BackgroundJob/PatronImport.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #18 from Brendan Lawlor <blawlor@clamsnet.org> --- This is awesome! I used to have run imp[orts in small batches, and even then I would often have to run SQL reports to verify that patrons were successfully imported and find any issues. For good measure I tested with a larger file with over 1,000 lines and also with a patron_attributes field with various values like: "SCODE:AVIATION,BCSI:Upload,REGQUA:ALLPATRONS,STUDENTID:999999" Note on the test plan. With the provided file of 500 records, on steps 8 and 9 the count should be 500: 8) Re-import the same file with 'Overwrite cardnumber' unchecked. Confirm 'already_in_db' count is 500. 9) Re-import again with 'Overwrite cardnumber' checked. Confirm 'overwritten' count is 500. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200665|0 |1 is patch| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200660|0 |1 is obsolete| | Attachment #200661|0 |1 is obsolete| | Attachment #200662|0 |1 is obsolete| | Attachment #200663|0 |1 is obsolete| | Attachment #200664|0 |1 is obsolete| | Attachment #200665|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #21 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201110 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201110&action=edit Bug 34632: Add Koha::BackgroundJob::PatronImport Introduces a new background job class for patron imports. enqueue() reads the uploaded file content into the job_args and uses line count as job_size for progress tracking. process() reconstitutes the file handle from the stored content, delegates to Koha::Patrons::Import with step_callback wired to $self->step, handles patron list creation, stores the import results, and clears file_content from the stored data to free space. Also registers patron_import in Koha::BackgroundJob::core_types_to_classes so _derived_class can resolve the job correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #23 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201112 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201112&action=edit Bug 34632: Use PatronImport background job in import_borrowers import_borrowers.pl now reads the uploaded file into memory and calls Koha::BackgroundJob::PatronImport->enqueue() instead of running the import synchronously. On success the template shows a confirmation with a link to the background job viewer; on failure it shows an error alert. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #25 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201114 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201114&action=edit Bug 34632: [DO NOT PUSH] Patron import sample data To be used to upload at <staff_url>/cgi-bin/koha/tools/import_borrowers.pl Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #27 from Lisette Scheer <lisette@bywatersolutions.com> --- I tested this morning with a very large file (~17k) and it failed with no warnings as to why, either in the interface or the logs. "The import job could not be enqueued. Please try again." It gave the above error when I tried to start the import. The import was queued into the background job but both when I clicked through and in the job it just says "The job failed." Looking at the console I saw the api call was truncated around 4800 records. I made the file smaller (4602 records) and it imported fine. It needs a warning (file too big/long?) preferably in the interface so it's clear to users why it's failing to work. Also the import was enqueued, even if it failed immediately, so the first message is wrong. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|42643 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201114|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #29 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 201903 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201903&action=edit Bug 34632: (QA follow-up) Use Koha::Uploader for patron import files Large files were failing to enqueue/process because the whole CSV was embedded in the job's JSON args, hitting DB/broker size limits. This now uploads the file via Koha::Uploader first (same pattern as MARC staging) and only passes a small uploaded_file_id through the job, and fixes the enqueue-failure message to not claim nothing happened when a job was in fact created. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #31 from Lisette Scheer <lisette@bywatersolutions.com> --- Works great. It took a while for me to finish qa because I had some non-sample data I was testing. I'm really excited for this! Please add release notes! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #32 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 203213 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203213&action=edit Bug 34632: (QA follow-up) Use Koha::Uploader for patron import files Large files were failing to enqueue/process because the whole CSV was embedded in the job's JSON args, hitting DB/broker size limits. This now uploads the file via Koha::Uploader first (same pattern as MARC staging) and only passes a small uploaded_file_id through the job, and fixes the enqueue-failure message to not claim nothing happened when a job was in fact created. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | Text to go in the| |Patron importing now runs release notes| |as a background job. This | |update prevents server | |timeouts when uploading | |large files and provides | |real-time progress | |tracking. A detailed | |summary is generated upon | |completion, making bulk | |patron imports much more | |reliable and easier to | |manage. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #33 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks everyone! Pushed to main for 26.11! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #35 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- (In reply to Katrin Fischer from comment #34)

Does this affect the CLI script for loading patrons as well?

Hi Katrin, it does not. The CLI script works the same as before. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Pushed to main |Needs documenting --- Comment #37 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- enhancement, no backport for 26.05.x -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43616 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43616 [Bug 43616] Add core support for scheduled/automated patron import (RFC) -- You are receiving this mail because: You are watching all bug changes.