[Bug 30326] New: Cronjob that checks biblios for MARC errors
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Bug ID: 30326 Summary: Cronjob that checks biblios for MARC errors Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: aleisha@catalyst.net.nz QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz It would be useful to have a script that could be run regularly which reported on biblios with MARC errors. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |aleisha@catalyst.net.nz |ity.org | Change sponsored?|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 --- Comment #1 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 131994 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=131994&action=edit Bug 30326: check_marc_errors.pl script which prints MARC errors This new check_marc_errors.pl script uses the MARC::Lint package to check bibliographic records for MARC errors, and print any errors found. The script can check all biblios, or specified biblios. To test: 1. Run the script from your commandline. This should not be run from within your koha shell. Replace the --intranet parameter with your staff client URL. sudo koha-shell kohadev -c "misc/cronjobs/check_marc_errors.pl --intranet http://localhost:8081/" > /tmp/marc_errors.html 2. The above command will run the script for all biblios in your database and print the output to a file /tmp/marc_errors.html. View the file and confirm it has been populated as expected, with errors! cat /tmp/marc_errors.html 3. Run the script again, this time specify one or more biblionumbers. sudo koha-shell kohadev -c "misc/cronjobs/check_marc_errors.pl --bibnum 1 --bibnum 2 --bibnum 3 --intranet http://localhost:8081/" > /tmp/marc_errors.html 4. Confirm the file is populated as expected cat /tmp/marc_errors.html Sponsored-by: Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This new release notes| |check_marc_errors.pl script | |uses the MARC::Lint package | |to check bibliographic | |records for MARC errors, | |and print any errors found. | |The script can check all | |biblios, or specified | |biblios. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #2 from Owen Leonard <oleonard@myacpl.org> --- This works well but there is a warning from the QA tool: FAIL misc/cronjobs/check_marc_errors.pl FAIL critic Subroutine "new" called using indirect syntax at line 99, column 18. See page 349 of PBP. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #131994|0 |1 is obsolete| | --- Comment #3 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 132038 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132038&action=edit Bug 30326: check_marc_errors.pl script which prints MARC errors This new check_marc_errors.pl script uses the MARC::Lint package to check bibliographic records for MARC errors, and print any errors found. The script can check all biblios, or specified biblios. To test: 1. Run the script from your commandline. This should not be run from within your koha shell. Replace the --intranet parameter with your staff client URL. sudo koha-shell kohadev -c "misc/cronjobs/check_marc_errors.pl --intranet http://localhost:8081/" > /tmp/marc_errors.html 2. The above command will run the script for all biblios in your database and print the output to a file /tmp/marc_errors.html. View the file and confirm it has been populated as expected, with errors! cat /tmp/marc_errors.html 3. Run the script again, this time specify one or more biblionumbers. sudo koha-shell kohadev -c "misc/cronjobs/check_marc_errors.pl --bibnum 1 --bibnum 2 --bibnum 3 --intranet http://localhost:8081/" > /tmp/marc_errors.html 4. Confirm the file is populated as expected cat /tmp/marc_errors.html Sponsored-by: Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132038|0 |1 is obsolete| | --- Comment #4 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 132053 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132053&action=edit Bug 30326: check_marc_errors.pl script which prints MARC errors This new check_marc_errors.pl script uses the MARC::Lint package to check bibliographic records for MARC errors, and print any errors found. The script can check all biblios, or specified biblios. To test: 1. Run the script from your commandline. This should not be run from within your koha shell. Replace the --intranet parameter with your staff client URL. sudo koha-shell kohadev -c "misc/cronjobs/check_marc_errors.pl --intranet http://localhost:8081/" > /tmp/marc_errors.html 2. The above command will run the script for all biblios in your database and print the output to a file /tmp/marc_errors.html. View the file and confirm it has been populated as expected, with errors! cat /tmp/marc_errors.html 3. Run the script again, this time specify one or more biblionumbers. sudo koha-shell kohadev -c "misc/cronjobs/check_marc_errors.pl --bibnum 1 --bibnum 2 --bibnum 3 --intranet http://localhost:8081/" > /tmp/marc_errors.html 4. Confirm the file is populated as expected cat /tmp/marc_errors.html Sponsored-by: Catalyst IT Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Hi, a couple remarks: 1. I'm not sure about the output, meaning that maybe as other scripts do, it could have an option switch for different formats. Not a blocker 2. In the --bibnum use case, it is not handling non-existing records correctly. It should report it correctly (right now it tells 'Must pass a MARC::Record object to check_record' which might be misleading for the end user). 3. I suggest you add a --where parameter, that is passed as a literal query to Koha::Biblios->search( $query, { prefetch => ['biblio_metadata'] } ); and use $biblio->metadata->record instead of GetMarcBiblio. Failing because of 2, primarily. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 --- Comment #6 from Aleisha Amohia <aleisha@catalyst.net.nz> --- (In reply to Tomás Cohen Arazi from comment #5)
3. I suggest you add a --where parameter, that is passed as a literal query to
Koha::Biblios->search( $query, { prefetch => ['biblio_metadata'] } );
and use $biblio->metadata->record instead of GetMarcBiblio.
I thought about doing it this way but it's significantly slower, not ideal when there are lots of biblios to check through. I can implement the other suggestions. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 --- Comment #7 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 133511 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133511&action=edit Bug 30326: (follow-up) Add output parameter and handle bad biblionumbers When the script is run without the --html parameter, the output will be simply text. When the script is run with the --html parameter, the output will display in an HTML table. Additional test plan: 5. Test running script with and without --html parameter and confirm output is correct. 6. Test running script with a --bibnum that does not exist and confirm the output acknowledges this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133511|0 |1 is obsolete| | --- Comment #8 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 133674 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133674&action=edit Bug 30326: (follow-up) Add output parameter and handle bad biblionumbers When the script is run without the --html parameter, the output will be simply text. When the script is run with the --html parameter, the output will display in an HTML table. Additional test plan: 5. Test running script with and without --html parameter and confirm output is correct. 6. Test running script with a --bibnum that does not exist and confirm the output acknowledges this. Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Aleisha, this works, but some questions: 1) Could we not use staffClientBaseURL instead of or as a default/fallback for the --intranet option? This way it would work for multiple instances without having differentiate the crontab entries. 2) The HTML output is a little hard to read. 2 suggestions: a) Add a header row b) Add <br> instead of line breaks. The output on the command line is something like: <td>100: Indicator 1 must be 0, 1 or 3 but it's " " 245: Indicator 1 must be 0 or 1 but it's " " 245: Indicator 2 must be 0 thru 9 but it's " " 245: Must end with . (period). 245: Non-filing indicator is non-numeric 245: First word, clouded, does not appear to be an article, check 2nd indicator ( ).</td> But then the lines all get "mushed together" into a long block of text without any separators making it hard to read/take apart: 100: Indicator 1 must be 0, 1 or 3 but it's " " 245: Indicator 1 must be 0 or 1 but it's " " 245: Indicator 2 must be 0 thru 9 but it's " " 245: Must end with . (period). 245: Non-filing indicator is non-numeric 245: First word, clouded, does not appear to be an article, check 2nd indicator ( ). Could you take a look and give me some feedback? Thx! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #9)
But then the lines all get "mushed together" into a long block of text without any separators making it hard to read/take apart:
Perhaps wrap it in "<pre></pre>" tags that preserve the original formatting from MARC::Lint? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 --- Comment #11 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 134345 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134345&action=edit Bug 30326: (follow-up) Tidy up output and use staffClientBaseURL -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I'll be happy to treat the changes as a QA follow-up (no new sign-off required), but could you update POD for the changed intranet option behaviour? Thx! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #134345|0 |1 is obsolete| | --- Comment #13 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 134447 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134447&action=edit Bug 30326: (follow-up) Tidy up output and use staffClientBaseURL -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com --- Comment #14 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Isn't it the same as misc/maintenance/search_for_data_inconsistencies.pl? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |In Discussion -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 Elizabeth John <elizabethjohn6654@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |elizabethjohn6654@gmail.com --- Comment #15 from Elizabeth John <elizabethjohn6654@gmail.com> --- The students may significantly face problem in academics and generating assignments, essays and other documents can take the services from the online platforms and online sites like https://dissertationwritinghelp.uk/buy-master-dissertation-service/ and many other sites. These services providing platforms are highly reliable and authentic. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30326 --- Comment #16 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Elizabeth John from comment #15) Spam -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org