[Bug 43525] New: Batch patron deletion and anonymization should be a background job
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Bug ID: 43525 Summary: Batch patron deletion and anonymization should be a background job Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: lucas@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Target Milestone: --- Patron deletion can be slow. When I test with 10,000 borrowers it takes 3minutes 45 seconds. We should move this to a background job. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #1 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205652 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205652&action=edit Bug 43525: Add a BatchDeletePatron background job and register it Patch from commit 7182384 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #2 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205664 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205664&action=edit Bug 43525: Change cleanborrowers to accept a background job Patch from commit 0c0cd0c -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |lucas@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #3 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- This is moving batch deletion into a background job. Now I am unsure if anonymization needs to be in a background job as well. It is not slow like patron deletion is. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #4 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205843 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205843&action=edit Bug 43525: Do not enqueue an empty job Noticed while attempting to add tests. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Medium patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #5 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205844 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205844&action=edit Bug 43525: Add unit tests prove -v t/db_dependent/Koha/BackgroundJob/BatchDeletePatron.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205652|0 |1 is obsolete| | Attachment #205664|0 |1 is obsolete| | Attachment #205843|0 |1 is obsolete| | Attachment #205844|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #6 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205845 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205845&action=edit Bug 43525: Add a BatchDeletePatron background job and register it To test: 0. APPLY PATCH and restart_all 1. Make sure you have some patron ready for deletion. I also check how many borrowers and deletedborrowers I have at this step: select count(*) from borrowers; select count(*) from deletedborrowers; 2. Go to Tools -> Batch patron deletion and anonymization 3. Enter in some params that will return you borrowers. who have not borrowed since, whose expiration date is before, and/or whose patron category is 4. Check 'Verify you want to delete patrons' and run the job 5. On the next step make note of how many borrowers should be deleted 6. Choose 'Permanently delete these patrons'. 7. A background job should be enqueued, when it is finished check borrowers again ( select count(*) from borrowers; ) 8. Also check deletedborrower ( select count(*) from deletedborrowers; ) 9. The borrower who were deleted should be gone for the borrowers table. Deleteborrowers should not have gained any new rows. 10. Now try all the steps again but at step 6 this time choose 'Move these patrons to the trash' 11. This time those borrowers should end up in the deletedborrowers table -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #7 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205846 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205846&action=edit Bug 43525: Change cleanborrowers to accept a background job Patch from commit 334b4da -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #8 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205847 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205847&action=edit Bug 43525: Do not enqueue an empty job Noticed while attempting to add tests. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #9 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 205848 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205848&action=edit Bug 43525: Add unit tests prove -v t/db_dependent/Koha/BackgroundJob/BatchDeletePatron.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43017 CC| |andrew@bywatersolutions.com Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43017 [Bug 43017] [OMNIBUS] Interface optimization -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=43523 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |26.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #10 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- When I click to see the progress of my deletion job I get: Exception 'Koha::Exception' thrown 'batch_patron_deletion is not a valid job_type' If I navigate to the Jobs list, I see this job has failed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #11 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- (In reply to Andrew Fuerste-Henry from comment #10)
When I click to see the progress of my deletion job I get: Exception 'Koha::Exception' thrown 'batch_patron_deletion is not a valid job_type'
If I navigate to the Jobs list, I see this job has failed.
User error; it's working for me now. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Michelle Spinney <mspinney@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mspinney@clamsnet.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205845|0 |1 is obsolete| | Attachment #205846|0 |1 is obsolete| | Attachment #205847|0 |1 is obsolete| | Attachment #205848|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #12 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 206681 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206681&action=edit Bug 43525: Add a BatchDeletePatron background job and register it To test: 0. APPLY PATCH and restart_all 1. Make sure you have some patron ready for deletion. I also check how many borrowers and deletedborrowers I have at this step: select count(*) from borrowers; select count(*) from deletedborrowers; 2. Go to Tools -> Batch patron deletion and anonymization 3. Enter in some params that will return you borrowers. who have not borrowed since, whose expiration date is before, and/or whose patron category is 4. Check 'Verify you want to delete patrons' and run the job 5. On the next step make note of how many borrowers should be deleted 6. Choose 'Permanently delete these patrons'. 7. A background job should be enqueued, when it is finished check borrowers again ( select count(*) from borrowers; ) 8. Also check deletedborrower ( select count(*) from deletedborrowers; ) 9. The borrower who were deleted should be gone for the borrowers table. Deleteborrowers should not have gained any new rows. 10. Now try all the steps again but at step 6 this time choose 'Move these patrons to the trash' 11. This time those borrowers should end up in the deletedborrowers table Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #13 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 206682 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206682&action=edit Bug 43525: Change cleanborrowers to accept a background job Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #14 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 206683 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206683&action=edit Bug 43525: Do not enqueue an empty job Noticed while attempting to add tests. Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #15 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 206684 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206684&action=edit Bug 43525: Add unit tests prove -v t/db_dependent/Koha/BackgroundJob/BatchDeletePatron.t Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |jacob.omara@openfifth.co.uk |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |zoho.roboto@bywatersolution | |s.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Jacob O'Mara <jacob.omara@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Jacob O'Mara <jacob.omara@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #206681|0 |1 is obsolete| | Attachment #206682|0 |1 is obsolete| | Attachment #206683|0 |1 is obsolete| | Attachment #206684|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #16 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208146 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208146&action=edit Bug 43525: Add a BatchDeletePatron background job and register it To test: 0. APPLY PATCH and restart_all 1. Make sure you have some patron ready for deletion. I also check how many borrowers and deletedborrowers I have at this step: select count(*) from borrowers; select count(*) from deletedborrowers; 2. Go to Tools -> Batch patron deletion and anonymization 3. Enter in some params that will return you borrowers. who have not borrowed since, whose expiration date is before, and/or whose patron category is 4. Check 'Verify you want to delete patrons' and run the job 5. On the next step make note of how many borrowers should be deleted 6. Choose 'Permanently delete these patrons'. 7. A background job should be enqueued, when it is finished check borrowers again ( select count(*) from borrowers; ) 8. Also check deletedborrower ( select count(*) from deletedborrowers; ) 9. The borrower who were deleted should be gone for the borrowers table. Deleteborrowers should not have gained any new rows. 10. Now try all the steps again but at step 6 this time choose 'Move these patrons to the trash' 11. This time those borrowers should end up in the deletedborrowers table Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #17 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208147 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208147&action=edit Bug 43525: Change cleanborrowers to accept a background job Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #18 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208148 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208148&action=edit Bug 43525: Do not enqueue an empty job Noticed while attempting to add tests. Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #19 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208149 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208149&action=edit Bug 43525: Add unit tests prove -v t/db_dependent/Koha/BackgroundJob/BatchDeletePatron.t Signed-off-by: Michelle Spinney <mspinney@clamsnet.org> Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #20 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208150 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208150&action=edit Bug 43525: (QA follow-up) Wrap patron deletion in a transaction Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #21 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208151 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208151&action=edit Bug 43525: (QA follow-up) Report patrons that are not found Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #22 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208152 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208152&action=edit Bug 43525: (QA follow-up) Show why a patron cannot be deleted Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 --- Comment #23 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Created attachment 208153 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208153&action=edit Bug 43525: (QA follow-up) Show a readable enqueue error Signed-off-by: Jacob O'Mara <jacob.omara@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_26_11_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43525 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |Enhancement moves the batch release notes| |deletion of patrons into a | |background job. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org