[Bug 34611] New: Add a script for pseudonymizing existing data
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Bug ID: 34611 Summary: Add a script for pseudonymizing existing data Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz Some libraries would like to eliminate old statistics, but keep the data for reporting. While pseudonymization works on new transactions once enabled, it would be nice to have a method to pseudonymize older transactions -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrewfh@dubcolib.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Emmi Takkinen <emmi.takkinen@koha-suomi.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emmi.takkinen@koha-suomi.fi --- Comment #1 from Emmi Takkinen <emmi.takkinen@koha-suomi.fi> --- +1 for this. This would simplify making statistics reports (no need to make join e.g. borrowers table to get patrons information). We also have plugin to collect statistics and using pseudonymized_transactions table with it currently means that we can't generate reports before pseudonymization was enabled. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 154808 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=154808&action=edit Bug 34611: Add pseudonymize routine to Koha::Statistics This patch adds a new routine to pseudonymize a statistic and adjusts C4::Stats to use this new routine. Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for the existence of objects before using them (in the case of older stats where things may have been deleted) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 154809 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=154809&action=edit Bug 34611: Add a cript to pseudonymize statistics from before a given date This script takes a date parameter in SQL format and pseudonymizes all statistics found before this date. Only values that can be found will be added, i.e. no deleted patron or item info will be present. Additionally - the values stored will be the current values from patrons and items, so some info will be approximate, much as it is when joining from the statistics table for reporting. To test: 1 - Perform some issues/returns/renewals/on-site checkouts 2 - Make sure Pseudonymization is disabled 3 - perl misc/maintenance/pseudonymize_statistics.pl 4 - Script ends and reports that preference is not active 5 - Enable the pref, and choose some borrower and item fields NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml 6 - perl misc/maintenance/pseudonymize_statistics.pl 7 - sudo koha-mysql kohadev 8 - SELECT * FROM pseudonymized_transactions 9 - Confirm data is correctly stored 10 - DELETE FROM pseudonymized_transactions; 11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00'; 12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59"; 13 - SELECT * FROM pseudonymized_transactions; 14 - Confirm no entries were made 15 - Select different options in Pseudonmyization prefs, including borrower attributes This wil require defining an attribute that can be kept for pseudonymization 16 - Confirm options are correctly pseudonymized -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Needs Unit tests -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 hebah@bywatersolutions.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hebah@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #154808|0 |1 is obsolete| | Attachment #154809|0 |1 is obsolete| | --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 157753 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157753&action=edit Bug 34611: Unit tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #6 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 157754 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157754&action=edit Bug 34611: Add pseudonymize routine to Koha::Statistics This patch adds a new routine to pseudonymize a statistic and adjusts C4::Stats to use this new routine. Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for the existence of objects before using them (in the case of older stats where things may have been deleted) Tests are added and can be run using: 1. Run: $ ktd --shell k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \ t/db_dependent/Koha/Pseudonymization.t => SUCCESS: New tests pass, old tests keep passing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #7 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 157755 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157755&action=edit Bug 34611: Add a script to pseudonymize statistics from before a given date This script takes a date parameter in SQL format and pseudonymizes all statistics found before this date. Only values that can be found will be added, i.e. no deleted patron or item info will be present. Additionally - the values stored will be the current values from patrons and items, so some info will be approximate, much as it is when joining from the statistics table for reporting. To test: 1 - Perform some issues/returns/renewals/on-site checkouts 2 - Make sure Pseudonymization is disabled 3 - perl misc/maintenance/pseudonymize_statistics.pl 4 - Script ends and reports that preference is not active 5 - Enable the pref, and choose some borrower and item fields NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml 6 - perl misc/maintenance/pseudonymize_statistics.pl 7 - sudo koha-mysql kohadev 8 - SELECT * FROM pseudonymized_transactions 9 - Confirm data is correctly stored 10 - DELETE FROM pseudonymized_transactions; 11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00'; 12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59"; 13 - SELECT * FROM pseudonymized_transactions; 14 - Confirm no entries were made 15 - Select different options in Pseudonmyization prefs, including borrower attributes This wil require defining an attribute that can be kept for pseudonymization 16 - Confirm options are correctly pseudonymized -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|tomascohen@gmail.com |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157753|0 |1 is obsolete| | --- Comment #8 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- Created attachment 157763 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157763&action=edit Bug 34611: Unit tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157754|0 |1 is obsolete| | --- Comment #9 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- Created attachment 157764 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157764&action=edit Bug 34611: Add pseudonymize routine to Koha::Statistics This patch adds a new routine to pseudonymize a statistic and adjusts C4::Stats to use this new routine. Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for the existence of objects before using them (in the case of older stats where things may have been deleted) Tests are added and can be run using: 1. Run: $ ktd --shell k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \ t/db_dependent/Koha/Pseudonymization.t => SUCCESS: New tests pass, old tests keep passing Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157755|0 |1 is obsolete| | --- Comment #10 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- Created attachment 157765 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157765&action=edit Bug 34611: Add a script to pseudonymize statistics from before a given date This script takes a date parameter in SQL format and pseudonymizes all statistics found before this date. Only values that can be found will be added, i.e. no deleted patron or item info will be present. Additionally - the values stored will be the current values from patrons and items, so some info will be approximate, much as it is when joining from the statistics table for reporting. To test: 1 - Perform some issues/returns/renewals/on-site checkouts 2 - Make sure Pseudonymization is disabled 3 - perl misc/maintenance/pseudonymize_statistics.pl 4 - Script ends and reports that preference is not active 5 - Enable the pref, and choose some borrower and item fields NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml 6 - perl misc/maintenance/pseudonymize_statistics.pl 7 - sudo koha-mysql kohadev 8 - SELECT * FROM pseudonymized_transactions 9 - Confirm data is correctly stored 10 - DELETE FROM pseudonymized_transactions; 11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00'; 12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59"; 13 - SELECT * FROM pseudonymized_transactions; 14 - Confirm no entries were made 15 - Select different options in Pseudonmyization prefs, including borrower attributes This wil require defining an attribute that can be kept for pseudonymization 16 - Confirm options are correctly pseudonymized Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #11 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- It's worth noting that running this script repeatedly *did* create duplicate entries in the pseudonymized_transactions table. I don't feel like that's a blocker, but should be documented. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #12 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Andrew Fuerste-Henry from comment #11)
It's worth noting that running this script repeatedly *did* create duplicate entries in the pseudonymized_transactions table. I don't feel like that's a blocker, but should be documented.
That's bad. Let's try to fix it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |33608 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- This will need some rebasing work on top of bug 33608, which was much longer in the queue and thus pushed first. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33608 [Bug 33608] Allow to get statistics about found/recovered books -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157763|0 |1 is obsolete| | Attachment #157764|0 |1 is obsolete| | Attachment #157765|0 |1 is obsolete| | --- Comment #14 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 157869 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157869&action=edit Bug 34611: Unit tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #15 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 157870 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157870&action=edit Bug 34611: Add pseudonymize routine to Koha::Statistics This patch adds a new routine to pseudonymize a statistic and adjusts C4::Stats to use this new routine. Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for the existence of objects before using them (in the case of older stats where things may have been deleted) Tests are added and can be run using: 1. Run: $ ktd --shell k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \ t/db_dependent/Koha/Pseudonymization.t => SUCCESS: New tests pass, old tests keep passing Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #16 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 157871 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157871&action=edit Bug 34611: Add a script to pseudonymize statistics from before a given date This script takes a date parameter in SQL format and pseudonymizes all statistics found before this date. Only values that can be found will be added, i.e. no deleted patron or item info will be present. Additionally - the values stored will be the current values from patrons and items, so some info will be approximate, much as it is when joining from the statistics table for reporting. To test: 1 - Perform some issues/returns/renewals/on-site checkouts 2 - Make sure Pseudonymization is disabled 3 - perl misc/maintenance/pseudonymize_statistics.pl 4 - Script ends and reports that preference is not active 5 - Enable the pref, and choose some borrower and item fields NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml 6 - perl misc/maintenance/pseudonymize_statistics.pl 7 - sudo koha-mysql kohadev 8 - SELECT * FROM pseudonymized_transactions 9 - Confirm data is correctly stored 10 - DELETE FROM pseudonymized_transactions; 11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00'; 12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59"; 13 - SELECT * FROM pseudonymized_transactions; 14 - Confirm no entries were made 15 - Select different options in Pseudonmyization prefs, including borrower attributes This wil require defining an attribute that can be kept for pseudonymization 16 - Confirm options are correctly pseudonymized Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #17 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 157872 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157872&action=edit Bug 34611: Confirm run if pseudonymized transactions found and add verbose feedback -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #18 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Tomás Cohen Arazi from comment #12)
(In reply to Andrew Fuerste-Henry from comment #11)
It's worth noting that running this script repeatedly *did* create duplicate entries in the pseudonymized_transactions table. I don't feel like that's a blocker, but should be documented.
That's bad. Let's try to fix it.
The whole point is that we shouldn't be able to get to the patron from the pseudonymized, so pointing out the overlap would seem wrong. It would also get tricky where one pseudonymized today might provide different patron info than an original, so detecting duplicates would be flawed. As a safeguard against accidental repetition I added a rudimentary check - if there are pseudonymized transactions found before the date you must now confirm the run or we exit. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157869|0 |1 is obsolete| | --- Comment #19 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- Created attachment 157877 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157877&action=edit Bug 34611: Unit tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157870|0 |1 is obsolete| | --- Comment #20 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- Created attachment 157878 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157878&action=edit Bug 34611: Add pseudonymize routine to Koha::Statistics This patch adds a new routine to pseudonymize a statistic and adjusts C4::Stats to use this new routine. Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for the existence of objects before using them (in the case of older stats where things may have been deleted) Tests are added and can be run using: 1. Run: $ ktd --shell k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \ t/db_dependent/Koha/Pseudonymization.t => SUCCESS: New tests pass, old tests keep passing Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157871|0 |1 is obsolete| | --- Comment #21 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- Created attachment 157879 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157879&action=edit Bug 34611: Add a script to pseudonymize statistics from before a given date This script takes a date parameter in SQL format and pseudonymizes all statistics found before this date. Only values that can be found will be added, i.e. no deleted patron or item info will be present. Additionally - the values stored will be the current values from patrons and items, so some info will be approximate, much as it is when joining from the statistics table for reporting. To test: 1 - Perform some issues/returns/renewals/on-site checkouts 2 - Make sure Pseudonymization is disabled 3 - perl misc/maintenance/pseudonymize_statistics.pl 4 - Script ends and reports that preference is not active 5 - Enable the pref, and choose some borrower and item fields NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml 6 - perl misc/maintenance/pseudonymize_statistics.pl 7 - sudo koha-mysql kohadev 8 - SELECT * FROM pseudonymized_transactions 9 - Confirm data is correctly stored 10 - DELETE FROM pseudonymized_transactions; 11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00'; 12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59"; 13 - SELECT * FROM pseudonymized_transactions; 14 - Confirm no entries were made 15 - Select different options in Pseudonmyization prefs, including borrower attributes This wil require defining an attribute that can be kept for pseudonymization 16 - Confirm options are correctly pseudonymized Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Andrew Fuerste-Henry <andrewfh@dubcolib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157872|0 |1 is obsolete| | --- Comment #22 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- Created attachment 157880 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157880&action=edit Bug 34611: Confirm run if pseudonymized transactions found and add verbose feedback Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #23 from Andrew Fuerste-Henry <andrewfh@dubcolib.org> --- re-signed-off with new check and verbose mode! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #157877|0 |1 is obsolete| | Attachment #157878|0 |1 is obsolete| | Attachment #157879|0 |1 is obsolete| | Attachment #157880|0 |1 is obsolete| | --- Comment #24 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 159909 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159909&action=edit Bug 34611: Unit tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #25 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 159910 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159910&action=edit Bug 34611: Add pseudonymize routine to Koha::Statistics This patch adds a new routine to pseudonymize a statistic and adjusts C4::Stats to use this new routine. Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for the existence of objects before using them (in the case of older stats where things may have been deleted) Tests are added and can be run using: 1. Run: $ ktd --shell k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \ t/db_dependent/Koha/Pseudonymization.t => SUCCESS: New tests pass, old tests keep passing Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #26 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 159911 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159911&action=edit Bug 34611: Add a script to pseudonymize statistics from before a given date This script takes a date parameter in SQL format and pseudonymizes all statistics found before this date. Only values that can be found will be added, i.e. no deleted patron or item info will be present. Additionally - the values stored will be the current values from patrons and items, so some info will be approximate, much as it is when joining from the statistics table for reporting. To test: 1 - Perform some issues/returns/renewals/on-site checkouts 2 - Make sure Pseudonymization is disabled 3 - perl misc/maintenance/pseudonymize_statistics.pl 4 - Script ends and reports that preference is not active 5 - Enable the pref, and choose some borrower and item fields NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml 6 - perl misc/maintenance/pseudonymize_statistics.pl 7 - sudo koha-mysql kohadev 8 - SELECT * FROM pseudonymized_transactions 9 - Confirm data is correctly stored 10 - DELETE FROM pseudonymized_transactions; 11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00'; 12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59"; 13 - SELECT * FROM pseudonymized_transactions; 14 - Confirm no entries were made 15 - Select different options in Pseudonmyization prefs, including borrower attributes This wil require defining an attribute that can be kept for pseudonymization 16 - Confirm options are correctly pseudonymized Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #27 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 159912 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159912&action=edit Bug 34611: Confirm run if pseudonymized transactions found and add verbose feedback Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Patch doesn't apply --- Comment #28 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- This also needs a rebase, conflict in Statistics.pm - I think it will be better if you resolve yourself. Also goes on my list to be pushed once it's bounced back. Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 34611: Unit tests Applying: Bug 34611: Add pseudonymize routine to Koha::Statistics Using index info to reconstruct a base tree... M Koha/Statistic.pm Falling back to patching base and 3-way merge... Auto-merging Koha/Statistic.pm CONFLICT (content): Merge conflict in Koha/Statistic.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 34611: Add pseudonymize routine to Koha::Statistics hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-34611-Add-pseudonymize-routine-to-KohaStatisti-tmb2y2g5.patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159909|0 |1 is obsolete| | Attachment #159910|0 |1 is obsolete| | Attachment #159911|0 |1 is obsolete| | Attachment #159912|0 |1 is obsolete| | --- Comment #29 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 163344 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163344&action=edit Bug 34611: Unit tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #30 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 163345 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163345&action=edit Bug 34611: Add pseudonymize routine to Koha::Statistics This patch adds a new routine to pseudonymize a statistic and adjusts C4::Stats to use this new routine. Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for the existence of objects before using them (in the case of older stats where things may have been deleted) Tests are added and can be run using: 1. Run: $ ktd --shell k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \ t/db_dependent/Koha/Pseudonymization.t => SUCCESS: New tests pass, old tests keep passing Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #31 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 163346 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163346&action=edit Bug 34611: Add a script to pseudonymize statistics from before a given date This script takes a date parameter in SQL format and pseudonymizes all statistics found before this date. Only values that can be found will be added, i.e. no deleted patron or item info will be present. Additionally - the values stored will be the current values from patrons and items, so some info will be approximate, much as it is when joining from the statistics table for reporting. To test: 1 - Perform some issues/returns/renewals/on-site checkouts 2 - Make sure Pseudonymization is disabled 3 - perl misc/maintenance/pseudonymize_statistics.pl 4 - Script ends and reports that preference is not active 5 - Enable the pref, and choose some borrower and item fields NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml 6 - perl misc/maintenance/pseudonymize_statistics.pl 7 - sudo koha-mysql kohadev 8 - SELECT * FROM pseudonymized_transactions 9 - Confirm data is correctly stored 10 - DELETE FROM pseudonymized_transactions; 11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00'; 12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59"; 13 - SELECT * FROM pseudonymized_transactions; 14 - Confirm no entries were made 15 - Select different options in Pseudonmyization prefs, including borrower attributes This wil require defining an attribute that can be kept for pseudonymization 16 - Confirm options are correctly pseudonymized Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #32 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 163347 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163347&action=edit Bug 34611: Confirm run if pseudonymized transactions found and add verbose feedback Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #33 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Looking at the script, I feel like adding a -c confirmation flag before any action is taken might be good. What do you think? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #34 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #35 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to Katrin Fischer from comment #33)
Looking at the script, I feel like adding a -c confirmation flag before any action is taken might be good. What do you think?
+1 scripts are always better with confirm flag -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #36 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 163756 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163756&action=edit Bug 34611: (follow-up) Add commit option to misc/maintenance/pseudonymize_statistics.pl To test: 1 - Perform some transactions 2 - Enable Pseudonymization 3 - perl misc/maintenance/pseudonymize_statistics.pl -v 4 - Confirm test run an nothing changed 5 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 6 - Confirm statistics are pseudonymized -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #37 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 163757 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163757&action=edit Bug 34611: (follow-up) Preserve statistic date if passed or null if not When testing the confirm option, I found that I was not warned on repeated runs. It turns out creating a new statistic object was setting the datetime to current timestamp This patch reset the object from the values passed To test: 1 - Disable Pseudonymization 2 - Perform some transactions, update datetimein DB to set them to yesterday 3 - Enable pseudonymization, ensure 'Date and time of transaction' is checked 4 - perl misc/maintenance/pseudonymize_statistics.pl -v 5 - Check the DB, note the new pseudonymized transactions have todays date 6 - DELETE FROM pseudonymized_transactions 7 - Apply patch 8 - perl misc/maintenance/pseudonymize_statistics.pl -v 9 - Check DB and confirm the transactoins have the original date 10 - DELETE FROM pseudonymized_transactions 11 - Unset date and time from pseudonymization pref 12 - perl misc/maintenance/pseudonymize_statistics.pl -v 13 - Confirm the transactions have a null date -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |ASSIGNED --- Comment #38 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Could we have QA on the last 2 patches please? Question: test plan for second patch doesn't mention the commit flag? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #39 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Katrin Fischer from comment #38)
Question: test plan for second patch doesn't mention the commit flag?
That's just forgetfulness :-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 --- Comment #40 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Nick Clemens from comment #39)
(In reply to Katrin Fischer from comment #38)
Question: test plan for second patch doesn't mention the commit flag?
That's just forgetfulness :-)
Thanks for clarifying ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This adds a new CLI script release notes| |`pseudonymize_statistics.pl | |` that will allow to | |pseudonymize rows in the | |statistics table before a | |given date. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #41 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I'm moving the follow-up patches to their own bug reports for clarity on the status of this bug. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37181 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37181 [Bug 37181] Add --confirm option switch to pseudonymize_statistics.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163756|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163757|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37182 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37182 [Bug 37182] 'datetime' field lost on pseudonymization -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting CC| |david@davidnind.com --- Comment #42 from David Nind <david@davidnind.com> --- Changing status to Needs documenting. (Bug was pushed to 24.05.00 and was not back ported to any earlier release.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |39301 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39301 [Bug 39301] pseudonymize_statistics.pl script generates too many background jobs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aude.charillon@openfifth.co | |.uk Documentation| |Aude Charillon contact| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/1170 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34611 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Needs documenting |RESOLVED --- Comment #43 from Aude Charillon <aude.charillon@openfifth.co.uk> --- Added to the Manual. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org