[Bug 32440] New: Allow selective deletion of statistics entries based on type by cleanup_database
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Bug ID: 32440 Summary: Allow selective deletion of statistics entries based on type by cleanup_database Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: andrewfh@dubcolib.org QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz Depends on: 24151 Cleanup_database can delete all statistics entries more than X days old. If one is using pseudonymization to create pseudonymized_transactions data, then one may wish to use that cleanup_database function to delete statistics entries that have been duplicated in pseudonymized_transactions. However, not all types of transactions in statistics are duplicated in pseudonymized transactions. Statistics includes checkouts, returns, renewals, payments, writeoffs, and localuse. Pseudonymized_transactions only includes checkouts, returns, and renewals. It would be helpful to be able to delete only statistics entries more than X days old and in a specified list of types. Approaching this from the other end, Bug 32438 suggests adding payments and writeoffs to pseudonymized_transactions. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24151 [Bug 24151] Add a pseudonymization process for patrons and transactions -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |19008 See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32438 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19008 [Bug 19008] Add more options to cleanup database script -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 --- Comment #1 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- *** Bug 32439 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Bug 32440 depends on bug 24151, which changed state. Bug 24151 Summary: Add a pseudonymization process for patrons and transactions https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24151 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |Needs documenting Resolution|FIXED |--- -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Bug 32440 depends on bug 24151, which changed state. Bug 24151 Summary: Add a pseudonymization process for patrons and transactions https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24151 What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 mathieu saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathsabypro@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Lari Taskula <lari.taskula@hypernova.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 --- Comment #2 from Lari Taskula <lari.taskula@hypernova.fi> --- Created attachment 172790 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172790&action=edit Bug 32440: Add missing unit test for Koha::Statistic->pseudonymize() To test: 1. prove t/db_dependent/Koha/Statistic.t Observe Subtest: pseudonymize() tests 1..6 ok 1 - some pseudonymization_types are defined ok 2 - Background job enqueued for type renew ok 3 - Background job enqueued for type issue ok 4 - Background job enqueued for type return ok 5 - Background job enqueued for type onsite_checkout ok 6 - Background job not enqueued for type not_gonna_pseudo ok 2 - pseudonymize() tests -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 --- Comment #3 from Lari Taskula <lari.taskula@hypernova.fi> --- Created attachment 172791 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172791&action=edit Bug 32440: Support purging statistics by type in cleanup_database.pl Cleanup_database can delete all statistics entries more than X days old. If one is using pseudonymization to create pseudonymized_transactions data, then one may wish to use that cleanup_database function to delete statistics entries that have been duplicated in pseudonymized_transactions. However, not all types of transactions in statistics are duplicated in pseudonymized transactions. Pseudonymized_transactions currently only includes checkouts, returns, and renewals. This patch adds two additional parameters to cleanup_database.pl: 1. --statistics-type Defines the types of statistics to purge. Will purge all types if parameter is omitted. Repeatable. 2. --statistics-type-pseudo Grabs values from @Koha::Statistic::pseudonymization_types. At the time of writing this patch, they are: renew, issue, return and onsite_checkout To test: 1. prove t/db_dependent/Koha/Statistic.t 2. Before applying this patch: 3. Create some statistics entry by checking out, renewing and checking in items. 4. perl misc/cronjobs/cleanup_database.pl --statistics 1 --verbose Observe: Purging statistics older than 1 days n statistics would have been removed Where n is the amount of statistics rows matching your test environment 5. Apply patch 6. Repeat step 4, observe same result 7. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --verbose Observe: Purging statistics older than 1 days with types "onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 8. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type test1 --statistics-type test2 --verbose Observe: Purging statistics older than 1 days with types "test1,test2". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 9. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --statistics-type test1 --verbose Observe: Purging statistics older than 1 days with types "test1,onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 10. Try previous cleanup_database.pl commands with --confirm flag and make sure correct rows are purged -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Lari Taskula <lari.taskula@hypernova.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lari.taskula@hypernova.fi Assignee|koha-bugs@lists.koha-commun |lari.taskula@hypernova.fi |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #172790|0 |1 is obsolete| | --- Comment #4 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 173874 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173874&action=edit Bug 32440: Add missing unit test for Koha::Statistic->pseudonymize() To test: 1. prove t/db_dependent/Koha/Statistic.t Observe Subtest: pseudonymize() tests 1..6 ok 1 - some pseudonymization_types are defined ok 2 - Background job enqueued for type renew ok 3 - Background job enqueued for type issue ok 4 - Background job enqueued for type return ok 5 - Background job enqueued for type onsite_checkout ok 6 - Background job not enqueued for type not_gonna_pseudo ok 2 - pseudonymize() tests Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #172791|0 |1 is obsolete| | --- Comment #5 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 173875 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173875&action=edit Bug 32440: Support purging statistics by type in cleanup_database.pl Cleanup_database can delete all statistics entries more than X days old. If one is using pseudonymization to create pseudonymized_transactions data, then one may wish to use that cleanup_database function to delete statistics entries that have been duplicated in pseudonymized_transactions. However, not all types of transactions in statistics are duplicated in pseudonymized transactions. Pseudonymized_transactions currently only includes checkouts, returns, and renewals. This patch adds two additional parameters to cleanup_database.pl: 1. --statistics-type Defines the types of statistics to purge. Will purge all types if parameter is omitted. Repeatable. 2. --statistics-type-pseudo Grabs values from @Koha::Statistic::pseudonymization_types. At the time of writing this patch, they are: renew, issue, return and onsite_checkout To test: 1. prove t/db_dependent/Koha/Statistic.t 2. Before applying this patch: 3. Create some statistics entry by checking out, renewing and checking in items. 4. perl misc/cronjobs/cleanup_database.pl --statistics 1 --verbose Observe: Purging statistics older than 1 days n statistics would have been removed Where n is the amount of statistics rows matching your test environment 5. Apply patch 6. Repeat step 4, observe same result 7. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --verbose Observe: Purging statistics older than 1 days with types "onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 8. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type test1 --statistics-type test2 --verbose Observe: Purging statistics older than 1 days with types "test1,test2". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 9. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --statistics-type test1 --verbose Observe: Purging statistics older than 1 days with types "test1,onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 10. Try previous cleanup_database.pl commands with --confirm flag and make sure correct rows are purged Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #173874|0 |1 is obsolete| | --- Comment #6 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 178467 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178467&action=edit Bug 32440: Add missing unit test for Koha::Statistic->pseudonymize() To test: 1. prove t/db_dependent/Koha/Statistic.t Observe Subtest: pseudonymize() tests 1..6 ok 1 - some pseudonymization_types are defined ok 2 - Background job enqueued for type renew ok 3 - Background job enqueued for type issue ok 4 - Background job enqueued for type return ok 5 - Background job enqueued for type onsite_checkout ok 6 - Background job not enqueued for type not_gonna_pseudo ok 2 - pseudonymize() tests Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #173875|0 |1 is obsolete| | --- Comment #7 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 178468 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178468&action=edit Bug 32440: Support purging statistics by type in cleanup_database.pl Cleanup_database can delete all statistics entries more than X days old. If one is using pseudonymization to create pseudonymized_transactions data, then one may wish to use that cleanup_database function to delete statistics entries that have been duplicated in pseudonymized_transactions. However, not all types of transactions in statistics are duplicated in pseudonymized transactions. Pseudonymized_transactions currently only includes checkouts, returns, and renewals. This patch adds two additional parameters to cleanup_database.pl: 1. --statistics-type Defines the types of statistics to purge. Will purge all types if parameter is omitted. Repeatable. 2. --statistics-type-pseudo Grabs values from @Koha::Statistic::pseudonymization_types. At the time of writing this patch, they are: renew, issue, return and onsite_checkout To test: 1. prove t/db_dependent/Koha/Statistic.t 2. Before applying this patch: 3. Create some statistics entry by checking out, renewing and checking in items. 4. perl misc/cronjobs/cleanup_database.pl --statistics 1 --verbose Observe: Purging statistics older than 1 days n statistics would have been removed Where n is the amount of statistics rows matching your test environment 5. Apply patch 6. Repeat step 4, observe same result 7. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --verbose Observe: Purging statistics older than 1 days with types "onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 8. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type test1 --statistics-type test2 --verbose Observe: Purging statistics older than 1 days with types "test1,test2". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 9. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --statistics-type test1 --verbose Observe: Purging statistics older than 1 days with types "test1,onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 10. Try previous cleanup_database.pl commands with --confirm flag and make sure correct rows are purged Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |25.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 25.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |Cleanup_database can delete release notes| |all statistics entries more | |than X days old. If one is | |using pseudonymization to | |create | |pseudonymized_transactions | |data, then one may wish to | |use that cleanup_database | |function to delete | |statistics entries that | |have been duplicated in | |pseudonymized_transactions. | |However, not all types of | |transactions in statistics | |are duplicated in | |pseudonymized transactions. | | | |Pseudonymized_transactions | |currently only includes | |checkouts, returns, | |and | |renewals. | | | |This patch adds | |two additional parameters | |to cleanup_database.pl: | | | |1. | |--statistics-type | | | |Defines | |the types of statistics to | |purge. Will purge all types | |if parameter is omitted. | |Repeatable. | | | |2. | |--statistics-type-pseudo | | | |Grabs values from | |@Koha::Statistic::pseudonym | |ization_types. At the time | |of implementation these | |are: renew, issue, return | |and onsite_checkout -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|Cleanup_database can delete |Cleanup_database can delete release notes|all statistics entries more |all statistics entries more |than X days old. If one is |than X days old. If one is |using pseudonymization to |using pseudonymization to |create |create |pseudonymized_transactions |pseudonymized_transactions |data, then one may wish to |data, then one may wish to |use that cleanup_database |use that cleanup_database |function to delete |function to delete |statistics entries that |statistics entries that |have been duplicated in |have been duplicated in |pseudonymized_transactions. |pseudonymized_transactions. |However, not all types of |However, not all types of |transactions in statistics |transactions in statistics |are duplicated in |are duplicated in |pseudonymized transactions. |pseudonymized transactions. | | |Pseudonymized_transactions |Pseudonymized_transactions |currently only includes |currently only includes |checkouts, returns, |checkouts, returns, |and |and |renewals. |renewals. | | |This patch adds |This patch adds |two additional parameters |two additional parameters |to cleanup_database.pl: |to cleanup_database.pl: | | |1. |1. |--statistics-type |`--statistics-type` | | |Defines |Defines the types of |the types of statistics to |statistics to purge. Will |purge. Will purge all types |purge all types if |if parameter is omitted. |parameter is omitted. |Repeatable. |Repeatable. | | |2. |2. |--statistics-type-pseudo |`--statistics-type-pseudo` | | |Grabs values from |Grabs values from |@Koha::Statistic::pseudonym |@Koha::Statistic::pseudonym |ization_types. At the time |ization_types. At the time |of implementation these |of implementation these |are: renew, issue, return |are: renew, issue, return |and onsite_checkout |and onsite_checkout -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Steve, OSLRI, USA <sspohn@oslri.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bywater@oslri.net, | |sspohn@oslri.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to main |Needs documenting --- Comment #9 from Fridolin Somers <fridolin.somers@biblibre.com> --- Enhancement not pushed to 24.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |39301 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 [Bug 39301] pseudonymize_statistics.pl script generates too many background jobs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aude.charillon@openfifth.co | |.uk Documentation| |Aude Charillon contact| | Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/1168 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32440 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED --- Comment #10 from Aude Charillon <aude.charillon@openfifth.co.uk> --- Added to the Koha Manual. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org