[Bug 34632] New: Patron Importing should be a background job
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Bug ID: 34632 Summary: Patron Importing should be a background job Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Patrons Assignee: koha-bugs@lists.koha-community.org Reporter: wizzyrea@gmail.com QA Contact: testopia@bugs.koha-community.org CC: gmcharlt@gmail.com, kyle.m.hall@gmail.com Hi friends, Now that we have a lovely queue for our background jobs, we should make the patron import use the queue to process those and store the outcome. This will help us avoid the 500/600/whatever you set apache timeouts experienced by libraries whenever they import batches of patrons of any significant size. Cheers, Liz -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Laura O'Neil <laura@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |laura@bywatersolutions.com --- Comment #1 from Laura O'Neil <laura@bywatersolutions.com> --- This would be a huge improvement for libraries who import large batches of patrons. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Valerie <valerie@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |valerie@bywatersolutions.co | |m --- Comment #2 from Valerie <valerie@bywatersolutions.com> --- I support this change! Currently, if a librarian imports a file too large, (over 2k), it eventually times out and gives an upstream proxy error, but the import continues chugging along invisibly. This is so confusing for the librarian, and they reach out for help or try multiple times. Sending it to the background jobs would make things much clearer and allow them to import their large patron files without it all falling apart. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 AspenCat Team <aspencatteam@clicweb.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aspencatteam@clicweb.org --- Comment #3 from AspenCat Team <aspencatteam@clicweb.org> --- We would support this move as well, it would result in fewer tickets to ByWater saving them time, and increase the turnaround time for our libraries to get data loaded. I am sure there are other library users and support companies in this situation who could also benefit. Ultimately this would be a cost savings investment for anyone wanting to take this development on. - Bob Bennhoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 trevor.diamond@mainlib.org changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |trevor.diamond@mainlib.org --- Comment #4 from trevor.diamond@mainlib.org --- Good idea. We need a similar bug to make patron batch modifications a background job too. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Initiative type|--- |Epic CC| |martin.renvoize@openfifth.c | |o.uk Strategic theme|--- |Performance Target Milestone|--- |26.11 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |pedro.amorim@openfifth.co.u |ity.org |k CC| |pedro.amorim@openfifth.co.u | |k -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #5 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Hi all, I will be working on this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #6 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200598 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200598&action=edit Bug 34632: Add step_callback support to Koha::Patrons::Import Adds an optional step_callback coderef parameter to import_patrons(). When provided it is called after every row — whether the row succeeds, is skipped as a duplicate, or is rejected as invalid — so callers can track progress. Also removes the 25-error cap that was there to protect browser response size; that constraint no longer applies when results are stored in a background job's data field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #7 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200599 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200599&action=edit Bug 34632: Add Koha::BackgroundJob::PatronImport Introduces a new background job class for patron imports. enqueue() reads the uploaded file content into the job_args and uses line count as job_size for progress tracking. process() reconstitutes the file handle from the stored content, delegates to Koha::Patrons::Import with step_callback wired to $self->step, handles patron list creation, stores the import results, and clears file_content from the stored data to free space. Also registers patron_import in Koha::BackgroundJob::core_types_to_classes so _derived_class can resolve the job correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #8 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200600 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200600&action=edit Bug 34632: Add tests for Koha::BackgroundJob::PatronImport Tests cover enqueue() (status, queue, job_size, job_type) and six process() scenarios: happy path, import errors (with validation that the 25-error cap has been removed), already_in_db and overwritten, patron list creation, and cancelled job handling. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #9 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200601 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200601&action=edit Bug 34632: Use PatronImport background job in import_borrowers import_borrowers.pl now reads the uploaded file into memory and calls Koha::BackgroundJob::PatronImport->enqueue() instead of running the import synchronously. On success the template shows a confirmation with a link to the background job viewer; on failure it shows an error alert. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #10 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200602 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200602&action=edit Bug 34632: Add patron_import background job display template patron_import.inc provides the report/detail/js blocks consumed by the background job viewer: a summary of counts when finished, a live progress bar while running, and the full feedback and error analysis in the detail section. background_jobs.tt gains a patron_import entry in the job type filter dropdown. Test plan: 1) Apply patches, restart plack. 2) Navigate to the patron import tool: <staff_url>/cgi-bin/koha/tools/import_borrowers.pl 3) Upload the provided sample CSV file and click 'Import patrons'. 4) Confirm the page shows "The patron import job has been enqueued!" with a link to view the job. 5) Follow the link to the background job. Confirm a progress bar is shown while the job is running. 6) Once finished, confirm the report shows the correct counts (imported, overwritten, already_in_db, invalid, total). 7) Confirm the imported patrons exist under 'Patrons'. 8) Re-import the same file with 'Overwrite cardnumber' unchecked. Confirm 'already_in_db' count is 5. 9) Re-import again with 'Overwrite cardnumber' checked. Confirm 'overwritten' count is 5. 10) Visit the background jobs admin and filter by type 'Import patrons': <staff_url>/cgi-bin/koha/admin/background_jobs.pl 11) Confirm the jobs appear in the list. 12) Run the tests: prove t/db_dependent/Koha/BackgroundJob/PatronImport.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #11 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200603 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200603&action=edit Bug 34632: [DO NOT PUSH] Patron import sample data To be used to upload at <staff_url>/cgi-bin/koha/tools/import_borrowers.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com | |, blawlor@clamsnet.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Brendan Lawlor <blawlor@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off Sponsorship status|--- |Sponsored Comma delimited| |CLAMS list of Sponsors| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Brendan Lawlor <blawlor@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200598|0 |1 is obsolete| | Attachment #200599|0 |1 is obsolete| | Attachment #200600|0 |1 is obsolete| | Attachment #200601|0 |1 is obsolete| | Attachment #200602|0 |1 is obsolete| | Attachment #200603|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #12 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200660 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200660&action=edit Bug 34632: Add step_callback support to Koha::Patrons::Import Adds an optional step_callback coderef parameter to import_patrons(). When provided it is called after every row — whether the row succeeds, is skipped as a duplicate, or is rejected as invalid — so callers can track progress. Also removes the 25-error cap that was there to protect browser response size; that constraint no longer applies when results are stored in a background job's data field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #13 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200661 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200661&action=edit Bug 34632: Add Koha::BackgroundJob::PatronImport Introduces a new background job class for patron imports. enqueue() reads the uploaded file content into the job_args and uses line count as job_size for progress tracking. process() reconstitutes the file handle from the stored content, delegates to Koha::Patrons::Import with step_callback wired to $self->step, handles patron list creation, stores the import results, and clears file_content from the stored data to free space. Also registers patron_import in Koha::BackgroundJob::core_types_to_classes so _derived_class can resolve the job correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #14 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200662 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200662&action=edit Bug 34632: Add tests for Koha::BackgroundJob::PatronImport Tests cover enqueue() (status, queue, job_size, job_type) and six process() scenarios: happy path, import errors (with validation that the 25-error cap has been removed), already_in_db and overwritten, patron list creation, and cancelled job handling. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #15 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200663 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200663&action=edit Bug 34632: Use PatronImport background job in import_borrowers import_borrowers.pl now reads the uploaded file into memory and calls Koha::BackgroundJob::PatronImport->enqueue() instead of running the import synchronously. On success the template shows a confirmation with a link to the background job viewer; on failure it shows an error alert. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #16 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200664 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200664&action=edit Bug 34632: Add patron_import background job display template patron_import.inc provides the report/detail/js blocks consumed by the background job viewer: a summary of counts when finished, a live progress bar while running, and the full feedback and error analysis in the detail section. background_jobs.tt gains a patron_import entry in the job type filter dropdown. Test plan: 1) Apply patches, restart plack. 2) Navigate to the patron import tool: <staff_url>/cgi-bin/koha/tools/import_borrowers.pl 3) Upload the provided sample CSV file and click 'Import patrons'. 4) Confirm the page shows "The patron import job has been enqueued!" with a link to view the job. 5) Follow the link to the background job. Confirm a progress bar is shown while the job is running. 6) Once finished, confirm the report shows the correct counts (imported, overwritten, already_in_db, invalid, total). 7) Confirm the imported patrons exist under 'Patrons'. 8) Re-import the same file with 'Overwrite cardnumber' unchecked. Confirm 'already_in_db' count is 5. 9) Re-import again with 'Overwrite cardnumber' checked. Confirm 'overwritten' count is 5. 10) Visit the background jobs admin and filter by type 'Import patrons': <staff_url>/cgi-bin/koha/admin/background_jobs.pl 11) Confirm the jobs appear in the list. 12) Run the tests: prove t/db_dependent/Koha/BackgroundJob/PatronImport.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #17 from Brendan Lawlor <blawlor@clamsnet.org> --- Created attachment 200665 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200665&action=edit Bug 34632: [DO NOT PUSH] Patron import sample data To be used to upload at <staff_url>/cgi-bin/koha/tools/import_borrowers.pl Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #18 from Brendan Lawlor <blawlor@clamsnet.org> --- This is awesome! I used to have run imp[orts in small batches, and even then I would often have to run SQL reports to verify that patrons were successfully imported and find any issues. For good measure I tested with a larger file with over 1,000 lines and also with a patron_attributes field with various values like: "SCODE:AVIATION,BCSI:Upload,REGQUA:ALLPATRONS,STUDENTID:999999" Note on the test plan. With the provided file of 500 records, on steps 8 and 9 the count should be 500: 8) Re-import the same file with 'Overwrite cardnumber' unchecked. Confirm 'already_in_db' count is 500. 9) Re-import again with 'Overwrite cardnumber' checked. Confirm 'overwritten' count is 500. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m Attachment #200665|1 |0 is patch| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200665|0 |1 is patch| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |lisette@bywatersolutions.co |y.org |m Status|Signed Off |Passed QA --- Comment #19 from Lisette Scheer <lisette@bywatersolutions.com> --- Works great. I tried with a large file as well with no problems. I did have one file that had an invalid column that didn't have a helpful error message, but I think that's out of scope so I'm not worrying about it here. Very excited for this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200660|0 |1 is obsolete| | Attachment #200661|0 |1 is obsolete| | Attachment #200662|0 |1 is obsolete| | Attachment #200663|0 |1 is obsolete| | Attachment #200664|0 |1 is obsolete| | Attachment #200665|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #20 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201109 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201109&action=edit Bug 34632: Add step_callback support to Koha::Patrons::Import Adds an optional step_callback coderef parameter to import_patrons(). When provided it is called after every row — whether the row succeeds, is skipped as a duplicate, or is rejected as invalid — so callers can track progress. Also removes the 25-error cap that was there to protect browser response size; that constraint no longer applies when results are stored in a background job's data field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #21 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201110 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201110&action=edit Bug 34632: Add Koha::BackgroundJob::PatronImport Introduces a new background job class for patron imports. enqueue() reads the uploaded file content into the job_args and uses line count as job_size for progress tracking. process() reconstitutes the file handle from the stored content, delegates to Koha::Patrons::Import with step_callback wired to $self->step, handles patron list creation, stores the import results, and clears file_content from the stored data to free space. Also registers patron_import in Koha::BackgroundJob::core_types_to_classes so _derived_class can resolve the job correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #22 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201111 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201111&action=edit Bug 34632: Add tests for Koha::BackgroundJob::PatronImport Tests cover enqueue() (status, queue, job_size, job_type) and six process() scenarios: happy path, import errors (with validation that the 25-error cap has been removed), already_in_db and overwritten, patron list creation, and cancelled job handling. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #23 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201112 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201112&action=edit Bug 34632: Use PatronImport background job in import_borrowers import_borrowers.pl now reads the uploaded file into memory and calls Koha::BackgroundJob::PatronImport->enqueue() instead of running the import synchronously. On success the template shows a confirmation with a link to the background job viewer; on failure it shows an error alert. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #24 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201113 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201113&action=edit Bug 34632: Add patron_import background job display template patron_import.inc provides the report/detail/js blocks consumed by the background job viewer: a summary of counts when finished, a live progress bar while running, and the full feedback and error analysis in the detail section. background_jobs.tt gains a patron_import entry in the job type filter dropdown. Test plan: 1) Apply patches, restart plack. 2) Navigate to the patron import tool: <staff_url>/cgi-bin/koha/tools/import_borrowers.pl 3) Upload the provided sample CSV file and click 'Import patrons'. 4) Confirm the page shows "The patron import job has been enqueued!" with a link to view the job. 5) Follow the link to the background job. Confirm a progress bar is shown while the job is running. 6) Once finished, confirm the report shows the correct counts (imported, overwritten, already_in_db, invalid, total). 7) Confirm the imported patrons exist under 'Patrons'. 8) Re-import the same file with 'Overwrite cardnumber' unchecked. Confirm 'already_in_db' count is 5. 9) Re-import again with 'Overwrite cardnumber' checked. Confirm 'overwritten' count is 5. 10) Visit the background jobs admin and filter by type 'Import patrons': <staff_url>/cgi-bin/koha/admin/background_jobs.pl 11) Confirm the jobs appear in the list. 12) Run the tests: prove t/db_dependent/Koha/BackgroundJob/PatronImport.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #25 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201114 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201114&action=edit Bug 34632: [DO NOT PUSH] Patron import sample data To be used to upload at <staff_url>/cgi-bin/koha/tools/import_borrowers.pl Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |enhancement --- Comment #26 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Changing to enhancement. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #27 from Lisette Scheer <lisette@bywatersolutions.com> --- I tested this morning with a very large file (~17k) and it failed with no warnings as to why, either in the interface or the logs. "The import job could not be enqueued. Please try again." It gave the above error when I tried to start the import. The import was queued into the background job but both when I clicked through and in the job it just says "The job failed." Looking at the console I saw the api call was truncated around 4800 records. I made the file smaller (4602 records) and it imported fine. It needs a warning (file too big/long?) preferably in the interface so it's clear to users why it's failing to work. Also the import was enqueued, even if it failed immediately, so the first message is wrong. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43016 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43016 [Bug 43016] [OMNIBUS] Server resource protection -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|42643 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201114|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #28 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 201902 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201902&action=edit Bug 34632: [DO NOT PUSH] Patron import sample data To be used to upload at <staff_url>/cgi-bin/koha/tools/import_borrowers.pl Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #29 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 201903 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201903&action=edit Bug 34632: (QA follow-up) Use Koha::Uploader for patron import files Large files were failing to enqueue/process because the whole CSV was embedded in the job's JSON args, hitting DB/broker size limits. This now uploads the file via Koha::Uploader first (same pattern as MARC staging) and only passes a small uploaded_file_id through the job, and fixes the enqueue-failure message to not claim nothing happened when a job was in fact created. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #30 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks Lisette, I've added a follow-up adopting the Koha::UploadedFiles core class which addressed the issue. I've also added a 17k patron csv file to the import sample data patch. I've tested myself and everything appears to be in order. Please have another go at it and let me know how it's working for you. Thank you. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #31 from Lisette Scheer <lisette@bywatersolutions.com> --- Works great. It took a while for me to finish qa because I had some non-sample data I was testing. I'm really excited for this! Please add release notes! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201903|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 --- Comment #32 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 203213 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203213&action=edit Bug 34632: (QA follow-up) Use Koha::Uploader for patron import files Large files were failing to enqueue/process because the whole CSV was embedded in the job's JSON args, hitting DB/broker size limits. This now uploads the file via Koha::Uploader first (same pattern as MARC staging) and only passes a small uploaded_file_id through the job, and fixes the enqueue-failure message to not claim nothing happened when a job was in fact created. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34632 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | Text to go in the| |Patron importing now runs release notes| |as a background job. This | |update prevents server | |timeouts when uploading | |large files and provides | |real-time progress | |tracking. A detailed | |summary is generated upon | |completion, making bulk | |patron imports much more | |reliable and easier to | |manage. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org