[Bug 39301] New: pseudonymize_statistics.pl script generates too many background jobs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Bug ID: 39301 Summary: pseudonymize_statistics.pl script generates too many background jobs Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz The script works, but in a large system it generates a background job per statistic, this can choke the workers and slow system processing. As this is intended as a one-time maintenance script I think we can remove the background-ing and pseudonymize the statistics directly -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 --- Comment #1 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 179158 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=179158&action=edit Bug 39301: pseudonymize_statistics.pl script generates too many background jobs This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job. Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search so we don't fetch any to pseudonymize that should not be. To test: 1 - Generate some entries in statistics (checkin, checkout, renew, etc) 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings) 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 4 - Statistics pseudonymized 5 - Check the background jobs table, note jobs were generated 6 - Apply patch 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 8 - You get same count as before, and are warned of existing pseudonymized transactions 9 - enter 'Y' to continue 10 - Check the pseudonymized_transacations table 11 - Confirm same rows were generated twice 12 - Check the background jobs 13 - Confirm there were no new jobs generated (for pseudonymization) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |34611 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 [Bug 34611] Add a script for pseudonymizing existing data -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #179158|0 |1 is obsolete| | --- Comment #2 from David Nind <david@davidnind.com> --- Created attachment 179177 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=179177&action=edit Bug 39301: pseudonymize_statistics.pl script generates too many background jobs This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job. Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search so we don't fetch any to pseudonymize that should not be. To test: 1 - Generate some entries in statistics (checkin, checkout, renew, etc) 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings) 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 4 - Statistics pseudonymized 5 - Check the background jobs table, note jobs were generated 6 - Apply patch 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 8 - You get same count as before, and are warned of existing pseudonymized transactions 9 - enter 'Y' to continue 10 - Check the pseudonymized_transacations table 11 - Confirm same rows were generated twice 12 - Check the background jobs 13 - Confirm there were no new jobs generated (for pseudonymization) Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #3 from David Nind <david@davidnind.com> --- Testing notes (using KTD): 1. Generate a password in KTD from the shell: htpasswd -bnBC 10 "" password | tr -d ':\n' | sed 's/$2y/$2a/' 2. Edit /etc/koha/sites/kohadev/koha-conf.xml 3. Add an entry before the </config>: <bcrypt_settings>whateverstep1generated</bcrypt_settings> 4. Restart everything: restart_all 5. Running the pseudonymize statistics script - if you get 0 statistics found, add the -b option with today's date, for example: perl misc/maintenance/pseudonymize_statistics.pl -b=2025-03-12 -v -c -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | CC| |lisette@bywatersolutions.co | |m -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39322 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |39322 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39322 [Bug 39322] When pseudonymizing don't fetch patron attributes if none are kept -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #179177|0 |1 is obsolete| | --- Comment #4 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 181155 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181155&action=edit Bug 39301: pseudonymize_statistics.pl script generates too many background jobs This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job. Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search so we don't fetch any to pseudonymize that should not be. To test: 1 - Generate some entries in statistics (checkin, checkout, renew, etc) 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings) 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 4 - Statistics pseudonymized 5 - Check the background jobs table, note jobs were generated 6 - Apply patch 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 8 - You get same count as before, and are warned of existing pseudonymized transactions 9 - enter 'Y' to continue 10 - Check the pseudonymized_transacations table 11 - Confirm same rows were generated twice 12 - Check the background jobs 13 - Confirm there were no new jobs generated (for pseudonymization) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 --- Comment #5 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- QA comment: This is a pragmatic workaround, but not a complete solution. The actual cause in Koha/Statistic is not addressed here. Enqueuing one record for this process is a bad idea. Too much overhead. We should run it on a result set in the plural object. And probably not run it directly after adding to statistics but on a daily basis or so. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |25.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 25.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32440 CC| |fridolin.somers@biblibre.co | |m --- Comment #7 from Fridolin Somers <fridolin.somers@biblibre.com> --- I see is uses @Koha::Statistic::pseudonymization_types from Bug 32440 Please provide a patch for 24.11 if you can Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 [Bug 32440] Allow selective deletion of statistics entries based on type by cleanup_database -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting --- Comment #8 from Fridolin Somers <fridolin.somers@biblibre.com> --- Since this is only optimization, lets skip 24.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Bug 39301 depends on bug 32440, which changed state. Bug 32440 Summary: Allow selective deletion of statistics entries based on type by cleanup_database https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED CC| |aude.charillon@openfifth.co | |.uk Resolution|--- |FIXED --- Comment #9 from Aude Charillon <aude.charillon@openfifth.co.uk> --- I don't think it is necessary to mention this in the Manual. Marking resolved. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Bug 39301 depends on bug 34611, which changed state. Bug 34611 Summary: Add a script for pseudonymizing existing data https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|25.05.00 |25.05.00,24.11.17 released in| | --- Comment #10 from Fridolin Somers <fridolin.somers@biblibre.com> --- Ah in the end I see it is in 24.11.08 but it is broken :
I see is uses @Koha::Statistic::pseudonymization_types from Bug 32440
I've added a fix for this, pushed in 24.11.17 : Bug 39301: (RMaint follow-up) Fix missing var Koha::Statistic::pseudonymization_types -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org