[Bug 31143] New: We should attempt to fix/identify all cases where '0000-00-00' may still remain in the database
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Bug ID: 31143 Summary: We should attempt to fix/identify all cases where '0000-00-00' may still remain in the database Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Database Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@ptfs-europe.com QA Contact: testopia@bugs.koha-community.org With the move to more and more of Koha using the REST API's and our rest API's being suitably strict about date formats, we need to retrospectively correct occurrences of 0000-00-00 dates appear in database fields system wide. I've had enough occurrences of bad data still in the database after a number of upgrade and it biting me in the UI that I think we should now look at introducing a script to either bulk fix these or at the least identify them and warn they will cause serious issues going forward. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |18016, 28440 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18016 [Bug 18016] C4::Auth_with_ldap::ldap_entry_2_hash inserts 0000-00-00 for invalid dateofbirth https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28440 [Bug 28440] Inventory tool tries to put a time into DateLastSeen -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |wainuiwitikapark@catalyst.n |ity.org |et.nz Status|NEW |ASSIGNED CC| |wainuiwitikapark@catalyst.n | |et.nz -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #1 from wainuiwitikapark@catalyst.net.nz --- Created attachment 154299 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=154299&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed 2) Run the fix_patron_dates.pl (-c -v) 3) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Laura Escamilla <Laura.escamilla@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |Laura.escamilla@bywatersolu | |tions.com --- Comment #2 from Laura Escamilla <Laura.escamilla@bywatersolutions.com> --- Hi! Thank you for working on this patch. I ran into this error: Missing right curly or square bracket at misc/cronjobs/fix_patron_dates.pl line 53, at end of line syntax error at misc/cronjobs/fix_patron_dates.pl line 53, at EOF After adding the curly this works as described! I'm marking this as Failed QA for now, but once the patch is updated I'll be happy to sign off! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #154299|0 |1 is obsolete| | --- Comment #3 from wainuiwitikapark@catalyst.net.nz --- Created attachment 154399 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=154399&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed 2) Run the fix_patron_dates.pl (-c -v) 3) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #4 from wainuiwitikapark@catalyst.net.nz --- (In reply to Laura Escamilla from comment #2)
Hi! Thank you for working on this patch. I ran into this error:
Missing right curly or square bracket at misc/cronjobs/fix_patron_dates.pl line 53, at end of line syntax error at misc/cronjobs/fix_patron_dates.pl line 53, at EOF
After adding the curly this works as described! I'm marking this as Failed QA for now, but once the patch is updated I'll be happy to sign off!
Thanks Laura for pointing that out. I have added the curly bracket to the end of the file and reuploaded the patch :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #154399|0 |1 is obsolete| | --- Comment #5 from David Nind <david@davidnind.com> --- Created attachment 155964 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155964&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed 2) Run the fix_patron_dates.pl (-c -v) 3) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This new script identifies release notes| |patron fields that have | |'0000-00-00' for date | |fields (for example: | |dateofbirth) and updates | |them to 'NULL'. Date fields | |with a value of '000-00-00' | |can cause upgrade and API | |issues, stop the | |autocomplete patron search | |working, and generate a 500 | |internal server error issue | |with normal patron | |searching. CC| |david@davidnind.com --- Comment #6 from David Nind <david@davidnind.com> --- Testing notes (using KTD): 1. Before applying the patch, I updated a couple of patron records using this SQL (Mary (49) and Henry (19)): . koha-mysql kohadev . update borrowers set dateofbirth ="0000-00-00" where borrowernumber = "49"; . update borrowers set date_renewed ="0000-00-00" where borrowernumber = "19"; 2. Note that the values updated now have a value of 0000-00-00 (SQL: select * from borrowers where borrowernumber = 19;). 3. In the staff interface, go to patrons. Note that the autocomplete for the search in the header doesn't work, and that you get a 500 error if you use the search on the left-hand side: Something went wrong when loading the table. 500: Internal Server Error. Month out of range. 4. Apply the patch (and restart_all). 5. Run the new script: misc/cronjobs/fix_patron_dates.pl -c -v 6. Output is: 2 wrong dates found Some other notes: 1. If you run when there are no wrong dates, you get this (with the -c and -v options): Use of uninitialized value $count in concatenation (.) or string at misc/cronjobs/fix_patron_dates.pl line 53. wrong dates found 2. I'm not sure of the protocol, but for many scripts: - you can add the -h or --help and get quick help with a description, what the different options do, and some examples. - normally the help would have the options, like -v, etc (in the script currently it just has c or v) - sometimes with verbose options, it may list the things changed 3. These could probably be added as new bugs, if determined as warranted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com Status|Signed Off |Failed QA --- Comment #7 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- This has been done already. Are you sure it's not coming from migration/import instead? I would add a check in misc/maintenance/search_for_data_inconsistencies.pl instead. IMO it does not make sense to have such specific cronjobs. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #8 from Lucas Gass <lucas@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #7)
This has been done already. Are you sure it's not coming from migration/import instead?
Just curious where this has been dealt with already. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #9 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Lucas Gass from comment #8)
(In reply to Jonathan Druart from comment #7)
This has been done already. Are you sure it's not coming from migration/import instead?
Just curious where this has been dealt with already.
Search for "sanitize_zero_date" in updatedatabase.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Bug 31143 depends on bug 28440, which changed state. Bug 28440 Summary: Inventory tool tries to put a time into DateLastSeen https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28440 What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |WORKSFORME -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com, | |philippe.blouin@inlibro.com --- Comment #10 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- A lot of people seem to have had problems recently with 0000-00-00 dates... There were two different threads on the mailing list about the same thing in the last month. https://lists.katipo.co.nz/public/koha/2024-June/060716.html https://lists.katipo.co.nz/public/koha/2024-June/060745.html -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Caroline Cyr La Rose from comment #10)
A lot of people seem to have had problems recently with 0000-00-00 dates... There were two different threads on the mailing list about the same thing in the last month.
https://lists.katipo.co.nz/public/koha/2024-June/060716.html
https://lists.katipo.co.nz/public/koha/2024-June/060745.html
One bug we have identified is bug 36488 - date of birth during patron self registration. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #12 from Fridolin Somers <fridolin.somers@biblibre.com> --- Looks like there are more fields than "dateofbirth dateenrolled dateexpiry date_renewed" to check : +lastseen +updated_on +debarred
I would add a check in misc/maintenance/search_for_data_inconsistencies.pl instead. +1
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #13 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- IIUC Wainui's script also corrected the dates to NULL instead of 0000-00-00. Is this something we can do with search_for_data_inconsistencies? I seem to remember that it corrects item-level itemtypes when the record-level itemtype is present, so I'm guessing yes? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|wainuiwitikapark@catalyst.n |koha-bugs@lists.koha-commun |et.nz |ity.org --- Comment #14 from wainuiwitikapark@catalyst.net.nz --- Apologies I haven't been able to find time to address the comments on this bug report Removing myself as the assignee, if someone else wants to pick this up, or I can check back in a few days when I have some time to look at it -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |wainuiwitikapark@catalyst.n |ity.org |et.nz -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155964|0 |1 is obsolete| | --- Comment #15 from wainuiwitikapark@catalyst.net.nz --- Created attachment 168636 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168636&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #16 from Jonathan Druart <jonathan.druart@gmail.com> --- This script is designed to report data inconsistencies but it is not expected to fix them. If we want we should at least require a flag (--fix or whatever). But in that case... What about the other inconsistencies? I would suggest to search for 0000-00-00 then display the query to fix them in the hint part. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #17 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Caroline Cyr La Rose from comment #13)
IIUC Wainui's script also corrected the dates to NULL instead of 0000-00-00. Is this something we can do with search_for_data_inconsistencies?
I seem to remember that it corrects item-level itemtypes when the record-level itemtype is present, so I'm guessing yes?
No, I don't think it is doing that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #18 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Caroline Cyr La Rose from comment #13)
IIUC Wainui's script also corrected the dates to NULL instead of 0000-00-00. Is this something we can do with search_for_data_inconsistencies?
Yes, I don't think it is correct.
I seem to remember that it corrects item-level itemtypes when the record-level itemtype is present, so I'm guessing yes?
No, I don't think the script is modifying data for now. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #19 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Could/should we add this as a command-line option? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #20 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Katrin Fischer from comment #19)
Could/should we add this as a command-line option?
It would be nice if we could provide people with some help in fixing the issue as it has come up a lot and can only be fixed with SQL right now. A hint could work too, but might require more explanations. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #21 from David Nind <david@davidnind.com> --- I had a go at testing this yesterday (see my comments on how I tested previously #c6). (I used KTD and the sample data, with some SQL changes to make some patron records incorrect. However, for me it didn't fix anything for me or output a list of patrons that have data that needs correction. I was going to have another go today, but I'll leave it for the moment. I sort of liked how the previously proposed script worked. 8) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #22 from wainuiwitikapark@catalyst.net.nz --- Hi everyone,
It would be nice if we could provide people with some help in fixing the issue as it has come up a lot and can only be fixed with SQL right now.
Yeah I could really use some help please with getting this fix through as soon as possible, it is a huge problem with many of our clients since a recent minor point.
I would suggest to search for 0000-00-00 then display the query to fix them in the hint part.
We would prefer the fix automatically fixes the dates, and not just give a prompt. However, I see that my latest patch shouldn't change data in misc/maintenance/search_for_data_inconsistencies.pl
I sort of liked how the previously proposed script worked.
I agree, and would be happy to amend it with the extra fields, unless someone can get this working as soon as possible Thanks, Wainui and the Catalyst team -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|wainuiwitikapark@catalyst.n |koha-bugs@lists.koha-commun |et.nz |ity.org --- Comment #23 from wainuiwitikapark@catalyst.net.nz --- Unassigning myself if someone wants to pick it up? Otherwise I can work on it if I have some guidance :) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #24 from Jonathan Druart <jonathan.druart@gmail.com> --- We can have both. The search_for_inconsistencies maintenance script can point to the other script. Also I think the script to fix the dates should not contain "patrons" as we could want to reuse it to fix dates from other database tables. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #25 from Blou <philippe.blouin@inlibro.com> --- we have a task opened to make a plugin out of this script. Mostly as a "learn the ropes" for interns, but it has the advantage of being easily hackable to make it what we want. Like fix things when there's no doubt as to what must be done. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |wainuiwitikapark@catalyst.n |ity.org |et.nz -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #168636|0 |1 is obsolete| | --- Comment #26 from wainuiwitikapark@catalyst.net.nz --- Created attachment 168770 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168770&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #27 from David Nind <david@davidnind.com> --- Thanks Wainui! I had a go at testing. 1. For me, misc/maintenance/search_for_data_inconsistencies.pl does not show anything in the output when I run it - where there are incorrect dates. I'm not sure if this is because of the error it generates when running (see step 2 below) - this occurred both before and after the patch was applied. 2. If the new script is run just with -v, it just says "0 wrong dates found and fixed". I guess that is correct. But if you were running it before using the -c option, I think you would want to know that "X wrong dates were found" (or something similar). 3. On the plus side, the new script does fix 0000-00-00 dates! 8-) Updated testing notes (using KTD and the sample data): 1. Fix record 369 (Strk̆v̜elser illustreret af Jean Anderson dansk udgave ved Lis Engel [overst̆telse ved Jesper Langer]) so that the misc/maintenance/search_for_data_inconsistencies.pl script doesn't show MARC records in the sample data that needs fixing. 2. Note that before applying the patch, running the search for data inconsistencies script returns an error/warning message: misc/maintenance/search_for_data_inconsistencies.pl Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483. 3. I updated a couple of patron records using this SQL (Mary (49) and Henry (19)): . koha-mysql kohadev . update borrowers set dateofbirth ="0000-00-00" where borrowernumber = "49"; . update borrowers set date_renewed ="0000-00-00" where borrowernumber = "19"; 4. Note that the values updated now have a value of 0000-00-00 (SQL example: select * from borrowers where borrowernumber = 19;). 5. In the staff interface, go to patrons. Note that the autocomplete for the search in the header doesn't work, and that you get a 500 error if you use the search on the left-hand side: Something went wrong when loading the table. 500: Internal Server Error. Month out of range. 6. Run the search for data inconsistencies script: misc/maintenance/search_for_data_inconsistencies.pl NOTE: This shows no information for me about any problems with patron dates with 0000-00-00 7. Output from running the new script: - with just -v: misc/cronjobs/fix_invalid_dates.pl -v 0 wrong dates found and fixed - with -c -v: misc/cronjobs/fix_invalid_dates.pl -c -v 2 wrong dates found and fixed 8. Repeat step 5 - things should now work as expected. Some other notes: 1. The description in misc/maintenance/search_for_data_inconsistencies.pl needs updating to add: * Patrons with 0000-00-00 dates in fields dateofbirth, dateenrolled, dateexpiry, date_renewed, lastseen, updated_on debarred 2. I'm not sure of the protocol, but for many scripts: - you can add the -h or --help and get quick help with a description, what the different options do, and some examples. - normally the help would have the options, like -v, etc (in the script currently it just has c or v) - sometimes with verbose options, it may list the things changed 3. These could probably be added as new bugs, if determined as warranted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #168770|0 |1 is obsolete| | --- Comment #28 from wainuiwitikapark@catalyst.net.nz --- Created attachment 168775 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168775&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #168775|0 |1 is obsolete| | --- Comment #29 from wainuiwitikapark@catalyst.net.nz --- Created attachment 168776 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168776&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36027 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #30 from David Nind <david@davidnind.com> --- I get this error message after I have updated some patrons to have 0000-00-00 dates - both before and after the patch. misc/maintenance/search_for_data_inconsistencies.pl Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483. Without the patch and without any incorrect dates, the above message doesn't appear. After the patch and after the dates are fixed with the new script, I get (as expected): misc/maintenance/search_for_data_inconsistencies.pl -v == 0 invalid dates found == -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #31 from wainuiwitikapark@catalyst.net.nz --- (In reply to David Nind from comment #30)
I get this error message after I have updated some patrons to have 0000-00-00 dates - both before and after the patch.
misc/maintenance/search_for_data_inconsistencies.pl Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483.
Without the patch and without any incorrect dates, the above message doesn't appear.
After the patch and after the dates are fixed with the new script, I get (as expected):
misc/maintenance/search_for_data_inconsistencies.pl -v == 0 invalid dates found ==
Thanks for testing and for your notes! - I changed a date to 0000-00-00 - I ran misc/maintenance/search_for_data_inconsistencies.pl -> I don't get that error - I added patch - I changed a date to 0000-00-00 - I ran misc/maintenance/search_for_data_inconsistencies.pl -> I don't get that error -> I get, as expected: == 1 invalid dates found == You may change the date with script: misc/cronjobs/fix_invalid_dates.pl (-c -v) - I then ran ./misc/cronjobs/fix_invalid_dates.pl -v -> I got the following, as expected: 1 invalid dates found - I then ran ./misc/cronjobs/fix_invalid_dates.pl -c -v -> I got the following, as expected: 1 invalid dates found 1 invalid dates fixed - Running it again -> I got, as expected: 0 invalid dates found 0 invalid dates fixed - Then I checked the date in the db, and it has indeed been fixed (changed to NULL) So yeah, I can't seem to replicate your issues :/ -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #32 from wainuiwitikapark@catalyst.net.nz --- (In reply to wainuiwitikapark from comment #31)
(In reply to David Nind from comment #30)
I get this error message after I have updated some patrons to have 0000-00-00 dates - both before and after the patch.
misc/maintenance/search_for_data_inconsistencies.pl Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483.
Without the patch and without any incorrect dates, the above message doesn't appear.
After the patch and after the dates are fixed with the new script, I get (as expected):
misc/maintenance/search_for_data_inconsistencies.pl -v == 0 invalid dates found ==
Thanks for testing and for your notes!
- I changed a date to 0000-00-00 - I ran misc/maintenance/search_for_data_inconsistencies.pl -> I don't get that error
- I added patch - I changed a date to 0000-00-00 - I ran misc/maintenance/search_for_data_inconsistencies.pl -> I don't get that error -> I get, as expected: == 1 invalid dates found == You may change the date with script: misc/cronjobs/fix_invalid_dates.pl (-c -v) - I then ran ./misc/cronjobs/fix_invalid_dates.pl -v -> I got the following, as expected: 1 invalid dates found - I then ran ./misc/cronjobs/fix_invalid_dates.pl -c -v -> I got the following, as expected: 1 invalid dates found 1 invalid dates fixed - Running it again -> I got, as expected: 0 invalid dates found 0 invalid dates fixed - Then I checked the date in the db, and it has indeed been fixed (changed to NULL)
So yeah, I can't seem to replicate your issues :/
Oh sorry I have retested and can now replicate the issue -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #33 from wainuiwitikapark@catalyst.net.nz --- (In reply to David Nind from comment #30)
I get this error message after I have updated some patrons to have 0000-00-00 dates - both before and after the patch.
misc/maintenance/search_for_data_inconsistencies.pl Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483.
Without the patch and without any incorrect dates, the above message doesn't appear.
After the patch and after the dates are fixed with the new script, I get (as expected):
misc/maintenance/search_for_data_inconsistencies.pl -v == 0 invalid dates found ==
What date fields were you changing? Were you changing dateofbirth? I have done some more testing and I think it is only dateofbirth that gives the error -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #34 from David Nind <david@davidnind.com> --- (In reply to wainuiwitikapark from comment #33)
What date fields were you changing? Were you changing dateofbirth?
I have done some more testing and I think it is only dateofbirth that gives the error
Thanks for checking and trying to replicate the issue I was having! I tested with changed dates for dateofbirth and date_renewed. The error happens with changing the dates before the patch is applied, so it must be something else already in the script. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #35 from wainuiwitikapark@catalyst.net.nz --- (In reply to David Nind from comment #34)
(In reply to wainuiwitikapark from comment #33)
What date fields were you changing? Were you changing dateofbirth?
I have done some more testing and I think it is only dateofbirth that gives the error
Thanks for checking and trying to replicate the issue I was having!
I tested with changed dates for dateofbirth and date_renewed.
The error happens with changing the dates before the patch is applied, so it must be something else already in the script.
No worries and thanks for clarifying that! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |P2 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #36 from wainuiwitikapark@catalyst.net.nz --- (In reply to David Nind from comment #30)
I get this error message after I have updated some patrons to have 0000-00-00 dates - both before and after the patch.
misc/maintenance/search_for_data_inconsistencies.pl Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483.
Without the patch and without any incorrect dates, the above message doesn't appear.
After the patch and after the dates are fixed with the new script, I get (as expected):
misc/maintenance/search_for_data_inconsistencies.pl -v == 0 invalid dates found ==
Hi David, Do you have any ideas on how I could potentially fix the time zone issue? Thanks :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #37 from David Nind <david@davidnind.com> --- (In reply to wainuiwitikapark from comment #36)
(In reply to David Nind from comment #30)
I get this error message after I have updated some patrons to have 0000-00-00 dates - both before and after the patch.
misc/maintenance/search_for_data_inconsistencies.pl Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483.
Without the patch and without any incorrect dates, the above message doesn't appear.
After the patch and after the dates are fixed with the new script, I get (as expected):
misc/maintenance/search_for_data_inconsistencies.pl -v == 0 invalid dates found ==
Hi David,
Do you have any ideas on how I could potentially fix the time zone issue?
Thanks :)
Sorry, I'm not a programmer and don't know Perl at all 8-(. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Paul Derscheid <paul.derscheid@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.derscheid@lmscloud.de --- Comment #38 from Paul Derscheid <paul.derscheid@lmscloud.de> --- I've looked into this a bit and this would either need: 1) dt_from_string to return a DateTime object instead of undef. 2) A fix for all dt_from_string calls within Koha::Patron. Pick your poison :D I'd like to hear some opinions. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #39 from Paul Derscheid <paul.derscheid@lmscloud.de> --- dt_from_string receiving something starting w/ '0000-0' that is. Koha/DateUtils.pm:62 `return if $date_string and $date_string =~ m|^0000-0|;` -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #40 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- Created attachment 170086 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170086&action=edit Bug 31143: Work around subtracting datetime if date_of_birth undef To test: 1) Run the following SQL: update borrowers set dateofbirth = '0000-00-00' where borrowernumber = 45; 2) Access the borrower edit page through staff UI: http://localhost:8081/cgi-bin/koha/members/memberentry.pl?op=edit_form&destination=circ&borrowernumber=45 3) Notice you get an error 4) Run the script: perl misc/maintenance/search_for_data_inconsistencies.pl 5) Notice you get an error 6) Apply patch restart plack and repeat -> no errors This is not perfect but its all I have time for, for now. Follow-ups are welcomed as always. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Pedro Amorim <pedro.amorim@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pedro.amorim@ptfs-europe.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #41 from Fridolin Somers <fridolin.somers@biblibre.com> ---
my $borrowers = Koha::Patrons->search(
Looks very strange to me this var name. We usually name it $patrons when it is a Koha::Patrons object. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #42 from Fridolin Somers <fridolin.somers@biblibre.com> ---
Work around subtracting datetime if date_of_birth undef This change will impact unit test 't/db_dependent/Koha/Patrons.t' no ?
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #43 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- (In reply to Fridolin Somers from comment #42)
Work around subtracting datetime if date_of_birth undef This change will impact unit test 't/db_dependent/Koha/Patrons.t' no ?
prove t/db_dependent/Koha/Patrons.t t/db_dependent/Koha/Patrons.t .. ok All tests successful. Files=1, Tests=43, 42 wallclock secs ( 0.06 usr 0.01 sys + 19.83 cusr 20.13 csys = 40.03 CPU) Result: PASS It doesn't seem to impact t/db_dependent/Koha/Patrons.t. I would expect it to impact it only if that test suite is already considering this possibility (undef $date_of_birth) which I don't believe it is, but I may be wrong. If the above is true, a new regression test for this should probably be added in t/db_dependent/Koha/Patrons.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #44 from wainuiwitikapark@catalyst.net.nz --- (In reply to Fridolin Somers from comment #41)
my $borrowers = Koha::Patrons->search(
Looks very strange to me this var name. We usually name it $patrons when it is a Koha::Patrons object.
I can change it if you think that's best? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P2 |P1 - high Assignee|wainuiwitikapark@catalyst.n |koha-bugs@lists.koha-commun |et.nz |ity.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Ray Delahunty <r.delahunty@arts.ac.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |r.delahunty@arts.ac.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Danyon Sewell <danyonsewell@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |danyonsewell@catalyst.net.n | |z --- Comment #45 from Danyon Sewell <danyonsewell@catalyst.net.nz> --- Kia ora Wainui I have just found that this issue doesn't just affect the borrowers table, I just had to fix this for a database where some of the items table also included the bad daterange, specifically the replacementpricedate column. It may be worth to include the columns within the items table in this script to better capture all of the possible places this bad daterange may end up.
From the items table the following columns could be affected:
dateaccessioned replacementpricedate datelastborrowed datelastseen damaged_on itemlost_on withdrawn_on deleted_on onloan Cheers, Danyon -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #46 from wainuiwitikapark@catalyst.net.nz --- (In reply to Danyon Sewell from comment #45)
Kia ora Wainui
I have just found that this issue doesn't just affect the borrowers table, I just had to fix this for a database where some of the items table also included the bad daterange, specifically the replacementpricedate column.
It may be worth to include the columns within the items table in this script to better capture all of the possible places this bad daterange may end up.
From the items table the following columns could be affected:
dateaccessioned replacementpricedate datelastborrowed datelastseen damaged_on itemlost_on withdrawn_on deleted_on onloan
Cheers, Danyon
Thanks Danyon! I'll look into that! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #168776|0 |1 is obsolete| | --- Comment #47 from David Nind <david@davidnind.com> --- Created attachment 170697 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170697&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170086|0 |1 is obsolete| | --- Comment #48 from David Nind <david@davidnind.com> --- Created attachment 170698 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170698&action=edit Bug 31143: Work around subtracting datetime if date_of_birth undef To test: 1) Run the following SQL: update borrowers set dateofbirth = '0000-00-00' where borrowernumber = 45; 2) Access the borrower edit page through staff UI: http://localhost:8081/cgi-bin/koha/members/memberentry.pl?op=edit_form&destination=circ&borrowernumber=45 3) Notice you get an error 4) Run the script: perl misc/maintenance/search_for_data_inconsistencies.pl 5) Notice you get an error 6) Apply patch restart plack and repeat -> no errors This is not perfect but its all I have time for, for now. Follow-ups are welcomed as always. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|This new script identifies |This enhancement: release notes|patron fields that have | |'0000-00-00' for date |1. |fields (for example: |Updates the |dateofbirth) and updates |misc/maintenance/search_for |them to 'NULL'. Date fields |_data_inconsistencies.pl |with a value of '000-00-00' |script so that it |can cause upgrade and API |identifies patrons with any |issues, stop the |date fields that have |autocomplete patron search |0000-00-00 values. |working, and generate a 500 | |internal server error issue |2. Adds |with normal patron |a new script |searching. |misc/cronjobs/fix_invalid_d | |ates.pl that fixes any | |patron date fields that | |have '0000-00-00' values | |(for example: dateofbirth) | |by updating them to 'NULL'. | |Patron date fields with a | |value of '000-00-00' can | |cause upgrade and API | |issues, stop the | |autocomplete patron search | |working, and generate a 500 | |internal server error issue | |with normal patron | |searching. --- Comment #49 from David Nind <david@davidnind.com> --- Testing notes (using KTD): 1. Fix record 369 (Strk̆v̜elser illustreret af Jean Anderson dansk udgave ved Lis Engel [overst̆telse ved Jesper Langer]) so that the misc/maintenance/search_for_data_inconsistencies.pl script doesn't show MARC records in the sample data that needs fixing. 2. Update a couple of patron records using this SQL (Mary (49) and Henry (19)): . koha-mysql kohadev . update borrowers set dateofbirth ="0000-00-00" where borrowernumber = "49"; . update borrowers set date_renewed ="0000-00-00" where borrowernumber = "19"; 3. Note that the values updated now have a value of 0000-00-00 (SQL example: select * from borrowers where borrowernumber = 19;). 4. In the staff interface, go to patrons. Note that the autocomplete for the search in the header doesn't work (search for hen), and that you get a 500 error if you use the search on the left-hand side: Something went wrong when loading the table. 500: Internal Server Error. Month out of range. 5. Run the search for data inconsistencies script: misc/maintenance/search_for_data_inconsistencies.pl. You will get this error: Can't call method "time_zone" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.36/DateTime.pm line 1483. 6. Apply the second patch: Bug 31143: Work around subtracting datetime if date_of_birth undef 7. Run this command: misc/maintenance/search_for_data_inconsistencies.pl -v Result: == Patrons with invalid age for category == * Patron borrowernumber=49 has an invalid age of 0 for their category 'PT' (18 to 999) => You may change the patron's category automatically with misc/cronjobs/update_patrons_category.pl 8. Apply the first patch. 9. Run this command again: misc/maintenance/search_for_data_inconsistencies.pl -v Result: now identifies that some patrons have invalid dates == Patrons with invalid age for category == * Patron borrowernumber=49 has an invalid age of 0 for their category 'PT' (18 to 999) => You may change the patron's category automatically with misc/cronjobs/update_patrons_category.pl == 2 invalid dates found == => You may change the date with script: misc/cronjobs/fix_invalid_dates.pl (-c -v) 10. Run the new script with just the -v option: misc/cronjobs/fix_invalid_dates.pl -v 2 invalid dates found 11. Run the new script with the -c -v options: misc/cronjobs/fix_invalid_dates.pl -c -v 2 wrong dates found and fixed 12. Repeat step 4 - things should now work as expected. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=37725 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=31093 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |wainuiwitikapark@catalyst.n |ity.org |et.nz -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170697|0 |1 is obsolete| | --- Comment #50 from wainuiwitikapark@catalyst.net.nz --- Created attachment 171812 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171812&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171812|0 |1 is obsolete| | --- Comment #51 from wainuiwitikapark@catalyst.net.nz --- Created attachment 171813 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171813&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171813|0 |1 is obsolete| | --- Comment #52 from wainuiwitikapark@catalyst.net.nz --- Created attachment 171814 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171814&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_invalid_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171814|0 |1 is obsolete| | --- Comment #53 from wainuiwitikapark@catalyst.net.nz --- Created attachment 171815 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171815&action=edit Bug 31143: Update patrons with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateofbirth dateenrolled dateexpiry date_renewed lastseen updated_on debarred 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_invalid_patron_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL Sponsored by: Toi Ohomai Institute of Technology, New Zealand and Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #54 from wainuiwitikapark@catalyst.net.nz --- Created attachment 171816 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171816&action=edit Bug 31143: Update items with dates of 0000-00-00 to NULL To test: 1) Change a patron's value of one or more of the following to 0000-00-00: dateaccessioned replacementpricedate datelastborrowed datelastseen damaged_on itemlost_on withdrawn_on deleted_on onloan 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_invalid_item_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171816|0 |1 is obsolete| | --- Comment #55 from wainuiwitikapark@catalyst.net.nz --- Created attachment 171817 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171817&action=edit Bug 31143: Update items with dates of 0000-00-00 to NULL To test: 1) Change an item's value of one or more of the following to 0000-00-00: dateaccessioned replacementpricedate datelastborrowed datelastseen damaged_on itemlost_on withdrawn_on deleted_on onloan 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run the fix_invalid_item_dates.pl (-c -v) 4) Notice the value has changed from 0000-00-00 to NULL -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff --- Comment #56 from wainuiwitikapark@catalyst.net.nz --- Hi I added in behaviour to fix incorrect item dates too -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #57 from Jonathan Druart <jonathan.druart@gmail.com> --- Hello Wainui, The title of this bug report states that the patches are supposed to "identify all cases in the database". But with those 2 patches we are only dealing with two tables: items and patrons. I am suggesting the following snippet of code that will allow you to catch invalid dates in the whole database: use Koha::Database; my $schema = Koha::Database->new->schema; # Loop over all the DBIx::Class classes for my $class ( sort values %{$schema->{class_mappings}} ) { # Retrieve the resultset so we can access the columns info my $rs = $schema->resultset($class); my $columns = $rs->result_source->columns_info; # Loop over the columns while ( my ( $column, $info ) = each %$columns ) { # Next if data type is not date/datetime/timestamp my $data_type = $info->{data_type}; next unless grep { $data_type =~ m{^$_$} } qw( timestamp datetime date ); # Count the invalid dates my $invalid_dates = $rs->search({ $column => '0000-00-00' })->count; next unless $invalid_dates; # "items.withdrawn_on contains 42 invalid dates" say sprintf "Column %s.%s contains %s invalid dates", $rs->result_source->name, $column, $invalid_dates; } } Let me know if you have any questions about it. I can help you to adjust it to your needs. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #58 from wainuiwitikapark@catalyst.net.nz --- (In reply to Jonathan Druart from comment #57)
Hello Wainui, The title of this bug report states that the patches are supposed to "identify all cases in the database". But with those 2 patches we are only dealing with two tables: items and patrons.
I am suggesting the following snippet of code that will allow you to catch invalid dates in the whole database:
use Koha::Database; my $schema = Koha::Database->new->schema; # Loop over all the DBIx::Class classes for my $class ( sort values %{$schema->{class_mappings}} ) { # Retrieve the resultset so we can access the columns info my $rs = $schema->resultset($class); my $columns = $rs->result_source->columns_info;
# Loop over the columns while ( my ( $column, $info ) = each %$columns ) { # Next if data type is not date/datetime/timestamp my $data_type = $info->{data_type}; next unless grep { $data_type =~ m{^$_$} } qw( timestamp datetime date );
# Count the invalid dates my $invalid_dates = $rs->search({ $column => '0000-00-00' })->count;
next unless $invalid_dates;
# "items.withdrawn_on contains 42 invalid dates" say sprintf "Column %s.%s contains %s invalid dates", $rs->result_source->name, $column, $invalid_dates; } }
Let me know if you have any questions about it. I can help you to adjust it to your needs.
Thanks Jonathan! I will have a go at implementing this -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #59 from wainuiwitikapark@catalyst.net.nz --- Created attachment 171918 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171918&action=edit Bug 31143: Fix/identify all cases where '0000-00-00' may still remain in the database This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL Test plan - Examples of setting up some 0000-00-00 date fields: e.g. Borrowers table update borrowers set dateexpiry="0000-00-00" where borrowernumber=49; e.g. Items table update items set datelastseen="0000-00-00" where itemnumber=12; To identify the cases: 1) Set up some 0000-00-00 date fields 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run fix_invalid_dates.pl -v To fix the cases: 1) Run fix_invalid_dates.pl -c -v 2) Notice the value has changed from 0000-00-00 to NULL e.g. Borrowers table select dateexpiry from borrowers where borrowernumber=49; e.g. Items table select datelastseen from items where itemnumber=12; -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171815|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171817|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Shi Yao Wang <shi-yao.wang@inLibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170698|0 |1 is obsolete| | --- Comment #60 from Shi Yao Wang <shi-yao.wang@inLibro.com> --- Created attachment 171976 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171976&action=edit Bug 31143: Work around subtracting datetime if date_of_birth undef To test: 1) Run the following SQL: update borrowers set dateofbirth = '0000-00-00' where borrowernumber = 45; 2) Access the borrower edit page through staff UI: http://localhost:8081/cgi-bin/koha/members/memberentry.pl?op=edit_form&destination=circ&borrowernumber=45 3) Notice you get an error 4) Run the script: perl misc/maintenance/search_for_data_inconsistencies.pl 5) Notice you get an error 6) Apply patch restart plack and repeat -> no errors This is not perfect but its all I have time for, for now. Follow-ups are welcomed as always. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Shi Yao Wang <shi-yao.wang@inLibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171918|0 |1 is obsolete| | --- Comment #61 from Shi Yao Wang <shi-yao.wang@inLibro.com> --- Created attachment 171977 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171977&action=edit Bug 31143: Fix/identify all cases where '0000-00-00' may still remain in the database This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL Test plan - Examples of setting up some 0000-00-00 date fields: e.g. Borrowers table update borrowers set dateexpiry="0000-00-00" where borrowernumber=49; e.g. Items table update items set datelastseen="0000-00-00" where itemnumber=12; To identify the cases: 1) Set up some 0000-00-00 date fields 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run fix_invalid_dates.pl -v To fix the cases: 1) Run fix_invalid_dates.pl -c -v 2) Notice the value has changed from 0000-00-00 to NULL e.g. Borrowers table select dateexpiry from borrowers where borrowernumber=49; e.g. Items table select datelastseen from items where itemnumber=12; Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Shi Yao Wang <shi-yao.wang@inLibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |shi-yao.wang@inLibro.com Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|This enhancement: |This enhancement: release notes| | |1. |1. |Updates the |Updates the |misc/maintenance/search_for |misc/maintenance/search_for |_data_inconsistencies.pl |_data_inconsistencies.pl |script so that it |script so that it |identifies patrons with any |identifies any occurrences |date fields that have |in the database with any |0000-00-00 values. |date fields that have | |0000-00-00 values. |2. Adds | |a new script |2. Adds |misc/cronjobs/fix_invalid_d |a new script |ates.pl that fixes any |misc/cronjobs/fix_invalid_d |patron date fields that |ates.pl that fixes any date |have '0000-00-00' values |fields that have |(for example: dateofbirth) |'0000-00-00' values by |by updating them to 'NULL'. |updating them to 'NULL'. |Patron date fields with a | |value of '000-00-00' can | |cause upgrade and API | |issues, stop the | |autocomplete patron search | |working, and generate a 500 | |internal server error issue | |with normal patron | |searching. | --- Comment #62 from wainuiwitikapark@catalyst.net.nz --- (In reply to Shi Yao Wang from comment #61)
Created attachment 171977 [details] [review] Bug 31143: Fix/identify all cases where '0000-00-00' may still remain in the database
This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL
Test plan -
Examples of setting up some 0000-00-00 date fields:
e.g. Borrowers table update borrowers set dateexpiry="0000-00-00" where borrowernumber=49;
e.g. Items table update items set datelastseen="0000-00-00" where itemnumber=12;
To identify the cases: 1) Set up some 0000-00-00 date fields 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run fix_invalid_dates.pl -v
To fix the cases: 1) Run fix_invalid_dates.pl -c -v 2) Notice the value has changed from 0000-00-00 to NULL
e.g. Borrowers table select dateexpiry from borrowers where borrowernumber=49;
e.g. Items table select datelastseen from items where itemnumber=12;
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Thanks so much for signing off! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171976|0 |1 is obsolete| | --- Comment #63 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172138 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172138&action=edit Bug 31143: Work around subtracting datetime if date_of_birth undef To test: 1) Run the following SQL: update borrowers set dateofbirth = '0000-00-00' where borrowernumber = 45; 2) Access the borrower edit page through staff UI: http://localhost:8081/cgi-bin/koha/members/memberentry.pl?op=edit_form&destination=circ&borrowernumber=45 3) Notice you get an error 4) Run the script: perl misc/maintenance/search_for_data_inconsistencies.pl 5) Notice you get an error 6) Apply patch restart plack and repeat -> no errors This is not perfect but its all I have time for, for now. Follow-ups are welcomed as always. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171977|0 |1 is obsolete| | --- Comment #64 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172139 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172139&action=edit Bug 31143: Fix/identify all cases where '0000-00-00' may still remain in the database This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL Test plan - Examples of setting up some 0000-00-00 date fields: e.g. Borrowers table update borrowers set dateexpiry="0000-00-00" where borrowernumber=49; e.g. Items table update items set datelastseen="0000-00-00" where itemnumber=12; To identify the cases: 1) Set up some 0000-00-00 date fields 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run fix_invalid_dates.pl -v To fix the cases: 1) Run fix_invalid_dates.pl -c -v 2) Notice the value has changed from 0000-00-00 to NULL e.g. Borrowers table select dateexpiry from borrowers where borrowernumber=49; e.g. Items table select datelastseen from items where itemnumber=12; Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #65 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172140 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172140&action=edit Bug 31143: (QA follow-up) move script to maintenance -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #66 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172141&action=edit Bug 31143: (QA follow-up) Tidy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #67 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- I moved the script from cronjobs to maintenance as I feel it should be avoided to be creating these, not expected to need fixing on a schedule, this can be dropped if desired. I also tidied. It would be nice to have increased verbosity and list which object were affected, but reporting the column allows for reporting, so this works. Nice patches, passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #172138|0 |1 is obsolete| | --- Comment #68 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172143&action=edit Bug 31143: Work around subtracting datetime if date_of_birth undef To test: 1) Run the following SQL: update borrowers set dateofbirth = '0000-00-00' where borrowernumber = 45; 2) Access the borrower edit page through staff UI: http://localhost:8081/cgi-bin/koha/members/memberentry.pl?op=edit_form&destination=circ&borrowernumber=45 3) Notice you get an error 4) Run the script: perl misc/maintenance/search_for_data_inconsistencies.pl 5) Notice you get an error 6) Apply patch restart plack and repeat -> no errors This is not perfect but its all I have time for, for now. Follow-ups are welcomed as always. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #172139|0 |1 is obsolete| | --- Comment #69 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172144 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172144&action=edit Bug 31143: Fix/identify all cases where '0000-00-00' may still remain in the database This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL Test plan - Examples of setting up some 0000-00-00 date fields: e.g. Borrowers table update borrowers set dateexpiry="0000-00-00" where borrowernumber=49; e.g. Items table update items set datelastseen="0000-00-00" where itemnumber=12; To identify the cases: 1) Set up some 0000-00-00 date fields 2) Run the misc/maintenance/search_for_data_inconsistencies.pl script 3) Run fix_invalid_dates.pl -v To fix the cases: 1) Run fix_invalid_dates.pl -c -v 2) Notice the value has changed from 0000-00-00 to NULL e.g. Borrowers table select dateexpiry from borrowers where borrowernumber=49; e.g. Items table select datelastseen from items where itemnumber=12; Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #172140|0 |1 is obsolete| | --- Comment #70 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172145 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172145&action=edit Bug 31143: (QA follow-up) move script to maintenance Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #172141|0 |1 is obsolete| | --- Comment #71 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172146 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172146&action=edit Bug 31143: (QA follow-up) Tidy Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|This enhancement: |This enhancement: release notes| | |1. |1. |Updates the |Updates the |misc/maintenance/search_for |misc/maintenance/search_for |_data_inconsistencies.pl |_data_inconsistencies.pl |script so that it |script so that it |identifies any occurrences |identifies any date fields |in the database with any |that have 0000-00-00 |date fields that have |values. |0000-00-00 values. | | |2. Adds a new |2. Adds |script |a new script |misc/maintenance/fix_invali |misc/cronjobs/fix_invalid_d |d_dates.pl that fixes any |ates.pl that fixes any date |date fields that have |fields that have |'0000-00-00' values (for |'0000-00-00' values by |example: dateofbirth) by |updating them to 'NULL'. |updating them to 'NULL'. | | | |Patron, item, and other | |date fields with a value of | |'000-00-00' can cause | |errors. This includes: | |- | |API errors | |- stopping the | |patron autocomplete search | |working | |- generating a 500 | |internal server error: | | . | |for normal patron searching | | . when displaying item | |data in the holdings table -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #72 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think given the amount of times we had this issue come up in chat and elsewhere, this will be much appreciated. Thanks for your perseverance Wainui! One small thing: I think there is no way to have the script tell you what it does. Could we add some kind of description about the purpose of the script to it? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |24.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #73 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #74 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- After pushing this I realized that Koha doesn't like borrowers.dateexpiry to be NULL :( The patron auto-complete search dies: Uncaught TypeError: item.expiry_date is null The error is the same if you use the auto-complete in the top search or when you transfer a list to another patron (found this while testing bug 30955) What I did was: * UPDATE borrowers set dateexpiry = "0000-00-00"; * Ran the script without -c to check that it would not change data * Then forgot I had not fixed the data and kept testing things... * I tested bug 30955, realized the problem and ran it with -c * auto-complete was still broken with the fixed data * I manually ran: UPDATE borrowers set dateexpiry = NULL; Still no luck. We either need to fix the auto-complete patron search or have the script re-calculate the dateexpiry from the patron category. I am leaning towards making the patron search deal with NULL. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #75 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Katrin Fischer from comment #74)
We either need to fix the auto-complete patron search or have the script re-calculate the dateexpiry from the patron category. I am leaning towards making the patron search deal with NULL.
borrowers.dateexpiry can be NULL so the code needs to be fixed. I have not tried it but this would be the idea: iff --git a/koha-tmpl/intranet-tmpl/prog/js/staff-global.js b/koha-tmpl/intranet-tmpl/prog/js/staff-global.js index 73a4a0251e1..28060fb35da 100644 --- a/koha-tmpl/intranet-tmpl/prog/js/staff-global.js +++ b/koha-tmpl/intranet-tmpl/prog/js/staff-global.js @@ -694,10 +694,13 @@ function patron_autocomplete(node, options) { } let new_date = new Date(); let today = Date.parse($date_to_rfc3339($date(new_date.toString()))); - let expiration = Date.parse( - $date_to_rfc3339($date(item.expiry_date.toString())) - ); - let expired = today > expiration; + let expired = false; + if (item.expiry_date){ + let expiration = Date.parse( + $date_to_rfc3339($date(item.expiry_date.toString())) + ); + expired = today > expiration; + } return $("<li></li>") .addClass(loggedInClass) .data("ui-autocomplete-item", item) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #76 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- (In reply to Katrin Fischer from comment #74)
We either need to fix the auto-complete patron search or have the script re-calculate the dateexpiry from the patron category. I am leaning towards making the patron search deal with NULL.
We have an installation currently where auto complete is erroring because a couple recently created patrons have updated_on as '0000-00-00'. I'm still investigating the cause (as I can't reproduce) but just adding this information here as well. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #77 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Pedro Amorim from comment #76)
(In reply to Katrin Fischer from comment #74)
We either need to fix the auto-complete patron search or have the script re-calculate the dateexpiry from the patron category. I am leaning towards making the patron search deal with NULL.
We have an installation currently where auto complete is erroring because a couple recently created patrons have updated_on as '0000-00-00'. I'm still investigating the cause (as I can't reproduce) but just adding this information here as well.
I have filed a new bug for the auto-complete issue: Bug 38254 - Patron auto-complete search explodes when dateexpiry is NULL -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=38254 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #78 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- (In reply to Katrin Fischer from comment #72)
I think given the amount of times we had this issue come up in chat and elsewhere, this will be much appreciated. Thanks for your perseverance Wainui!
Thanks Katrin!!
One small thing: I think there is no way to have the script tell you what it does. Could we add some kind of description about the purpose of the script to it?
Are you still wanting me to do this? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #79 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Wainui Witika-Park from comment #78)
(In reply to Katrin Fischer from comment #72)
I think given the amount of times we had this issue come up in chat and elsewhere, this will be much appreciated. Thanks for your perseverance Wainui!
Thanks Katrin!!
One small thing: I think there is no way to have the script tell you what it does. Could we add some kind of description about the purpose of the script to it?
Are you still wanting me to do this?
Yes please! I am leaving "additional_work_needed" as a keyword on anything with questions or a follow-up request. We moved the dateexpiry issue to another spot, but I think giving people a good clue about the script who didn't start out at the inconsistencies one would still be helpful. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #80 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- (In reply to Pedro Amorim from comment #76)
We have an installation currently where auto complete is erroring because a couple recently created patrons have updated_on as '0000-00-00'. I'm still investigating the cause (as I can't reproduce) but just adding this information here as well.
Leaving info here that this relates to bug 36822 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36822 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #81 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Created attachment 173900 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173900&action=edit Bug 31143: (Follow-up) Add description to fix_invalid_dates.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_24_05_candidate CC| |aleisha@catalyst.net.nz -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | --- Comment #82 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Sorry Wainui, I had missed the follow-up somehow! Pushed to main now. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_24_05_candidate | Version(s)|24.11.00 |24.11.00,24.05.06 released in| | Status|Pushed to main |Pushed to stable --- Comment #83 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Backported to 24.05.x for upcoming 24.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #84 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Lucas Gass (lukeg) from comment #83)
Backported to 24.05.x for upcoming 24.05.06
Please be aware of bug 38254 - currently we break auto-complete with the script, although it does the right thing. We need this fixed too. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #85 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> ---
Please be aware of bug 38254 - currently we break auto-complete with the script, although it does the right thing. We need this fixed too.
Thanks. Do you think I should revert this for now? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #86 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Lucas Gass (lukeg) from comment #85)
Please be aware of bug 38254 - currently we break auto-complete with the script, although it does the right thing. We need this fixed too.
Thanks. Do you think I should revert this for now?
it's broken patron search vs. broken auto-complete - both are not great. I would really prefer a patch for bug 38254 :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #87 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- If anyone else is interested: We have been setting up the cronjob in cron.daily/koha-custom #!/bin/sh sudo koha-foreach --enabled /usr/share/koha/bin/maintenance/fix_invalid_dates.pl -c -v -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.11.00,24.05.06 |24.11.00,24.05.06,23.11.11 released in| | Status|Pushed to stable |Pushed to oldstable --- Comment #88 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Aditya <adifbbk1@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |adifbbk1@gmail.com --- Comment #89 from Aditya <adifbbk1@gmail.com> --- I see nobody's talking about the error when the below line gets executed my $invalid_dates = $rs->search( { $column => '0000-00-00' } )->count; This is throwing the error as below: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Incorrect TIMESTAMP value: '0000-00-00' at ./fix_invalid_dates.pl line 65 I have made a quick fix for this. Please include to the next minor update (after the line:- my $schema = Koha::Database->new->schema;) if the similar is not implemented. # Disable strict mode for this session $schema->storage->dbh_do( sub { my ($storage, $dbh) = @_; $dbh->do("SET SESSION sql_mode = ''"); } ); -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #90 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Aditya from comment #89)
I see nobody's talking about the error when the below line gets executed
my $invalid_dates = $rs->search( { $column => '0000-00-00' } )->count;
This is throwing the error as below: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Incorrect TIMESTAMP value: '0000-00-00' at ./fix_invalid_dates.pl line 65
I have made a quick fix for this. Please include to the next minor update (after the line:- my $schema = Koha::Database->new->schema;) if the similar is not implemented.
# Disable strict mode for this session $schema->storage->dbh_do( sub { my ($storage, $dbh) = @_; $dbh->do("SET SESSION sql_mode = ''"); } );
Hi Aditya, since this is already in a release, we need a new bug number for the fix. Can you please file one and add a test plan? It might depend on DBMS version, so that might be why we didn't see that in testing. I remember we had a similar error. Please also add your DBMS and the version to the report! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 --- Comment #91 from Aditya <adifbbk1@gmail.com> --- (In reply to Katrin Fischer from comment #90)
(In reply to Aditya from comment #89)
I see nobody's talking about the error when the below line gets executed
my $invalid_dates = $rs->search( { $column => '0000-00-00' } )->count;
This is throwing the error as below: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Incorrect TIMESTAMP value: '0000-00-00' at ./fix_invalid_dates.pl line 65
I have made a quick fix for this. Please include to the next minor update (after the line:- my $schema = Koha::Database->new->schema;) if the similar is not implemented.
# Disable strict mode for this session $schema->storage->dbh_do( sub { my ($storage, $dbh) = @_; $dbh->do("SET SESSION sql_mode = ''"); } );
Hi Aditya, since this is already in a release, we need a new bug number for the fix. Can you please file one and add a test plan? It might depend on DBMS version, so that might be why we didn't see that in testing. I remember we had a similar error. Please also add your DBMS and the version to the report!
Please use bug number 38788 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |38788 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38788 [Bug 38788] Fix '0000-00-00' issue on database -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |Caroline Cyr La Rose contact| | Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/1037 --- Comment #92 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- I added this to the manual. Whoever is the last to check it for backport can resolve it. Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |RESOLVED Resolution|--- |FIXED --- Comment #93 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Applied to 22.11 for 22.11.28 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31143 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.11.00,24.05.06,23.11.11 |24.11.00,24.05.06,23.11.11, released in| |22.11.28 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org