[Bug 40777] New: 500 Error: Something went wrong when loading the table Should Exit Cleanly
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Bug ID: 40777 Summary: 500 Error: Something went wrong when loading the table Should Exit Cleanly Change sponsored?: --- Product: Koha Version: 24.11 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Cataloging Assignee: koha-bugs@lists.koha-community.org Reporter: jyorio@equinoxinitiative.org QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl It's possible this is related to/a duplicate of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40762 but I am not certain. In Koha 24.05 and 24.11, in the staff interface, if a bib with an item without a 952$a and 952$b is imported, when the staff member then goes to the bib record they receive a modal pop-up stating "Something went wrong when loading the table. 500: error". If they click OK on that message, a processing box sits on the screen (you can see the bib loaded behind it) and spins. The console log shows: datatables_24.1107000.js: 408 DataTables warning: table id=holdings_table - Ajax error. For more information about this error, please see https://datatables.net/tn/7 The "processing" box is not modal and it is possible to click on Edit, Manage items, and then edit the problem item to add the missing Home and Current libraries and resolve the cause of the error. This behavior does not occur in 23.11. While, of course, the item needs the Home and Current library, the error alert is a change in expected behavior from staff member perspective. A cleaner exit with a more informative message than a 500 error seems preferable. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #1 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 187526 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187526&action=edit Bug 40777: Introduce Koha::Database::DataInconsistency -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #2 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 187527 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187527&action=edit Bug 40777: Add 'Audit' button in the toolbar -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #3 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 187528 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187528&action=edit Bug 40777: Add info in the alert -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m, | |jonathan.druart@gmail.com, | |lucas@bywatersolutions.com, | |nick@bywatersolutions.com Status|NEW |In Discussion Assignee|koha-bugs@lists.koha-commun |jonathan.druart@gmail.com |ity.org | Version|24.11 |unspecified --- Comment #4 from Jonathan Druart <jonathan.druart@gmail.com> --- Patches for discussion and feedback. I am suggesting to move the code from the search_for_inconsistencies.pl maintenance script to a module to make it reusable from the Koha UI. Those patch moves only the first check (no valid home or holding library) to the module, and add an "Audit" icon to the bibliographic toolbar. https://snipboard.io/1owIgH.jpg If people are willing to test and provide feedback, I will provide more work. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Marion Durand <marion.durand@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marion.durand@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
From kohastructure `homebranch` varchar(10) DEFAULT NULL COMMENT 'foreign key from the branches
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #5 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- table for the library that owns this item (MARC21 952$a)', `holdingbranch` varchar(10) DEFAULT NULL COMMENT 'foreign key from the branches table for the library that is currently in possession item (MARC21 952$b)', While both columns may still be NULL, we will have such issues. What about NOT NULL to ensure integrity? If we create an item somewhere or import it, we should probably know where it belongs. Just suspecting that invalid branchcodes during import may be at stake. And note that the REST API who is called to fetch the items, may need some extra check while we dont address it in the table design? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #6 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Marcel de Rooy from comment #5)
From kohastructure `homebranch` varchar(10) DEFAULT NULL COMMENT 'foreign key from the branches table for the library that owns this item (MARC21 952$a)', `holdingbranch` varchar(10) DEFAULT NULL COMMENT 'foreign key from the branches table for the library that is currently in possession item (MARC21 952$b)',
While both columns may still be NULL, we will have such issues. What about NOT NULL to ensure integrity? If we create an item somewhere or import it, we should probably know where it belongs. Just suspecting that invalid branchcodes during import may be at stake.
And note that the REST API who is called to fetch the items, may need some extra check while we dont address it in the table design?
This is outside the scope. I have implemented the first test of misc/maintenance/search_for_data_inconsistencies.pl, but there are more: * incorrect item type (yes you could argue that a FK is missing as well for this one) * invalid marcxml * item fields in MARC * wrong authorised values * record without title The goal here is to offer the output of the script from the UI. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #7 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 188222 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188222&action=edit Bug 40777: Move all biblio tests to the new package -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #8 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 188223 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188223&action=edit Bug 40777: Add tests for invalid_item_library and no_item_type -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #9 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 188224 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188224&action=edit Bug 40777: Add tests for invalid_item_type -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #10 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 188225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188225&action=edit Bug 40777: Add tests for errors_in_marc -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #11 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 188226 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188226&action=edit Bug 40777: Add tests for nonexistent_AV -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #12 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 188227 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188227&action=edit Bug 40777: Add tests for empty_title -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #13 from Jonathan Druart <jonathan.druart@gmail.com> --- Ready for testing. The tests directly related to a bibliographic record have been moved to the new packages (which are now covered by unit tests). To test this you need to confirm that the UI shows (using the audit tool) the same errors/warnings we have with the maintenance script. I can provide a more detailed test plan if needed ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |phil@chetcolibrary.org --- Comment #14 from Phil Ringnalda <phil@chetcolibrary.org> --- The UI shows literally Errors found ARRAY(0xaaab121c8750) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #15 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 189598 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189598&action=edit Bug 40777: Flatten arrays Patch from commit e253e76 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #16 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Phil Ringnalda from comment #14)
The UI shows literally
Errors found ARRAY(0xaaab121c8750)
Oops sorry, it seems that I have not retested the UI after the follow-ups. Should be fixed now. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #187526|0 |1 is obsolete| | --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 190261 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190261&action=edit Bug 40777: Introduce Koha::Database::DataInconsistency Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #187527|0 |1 is obsolete| | --- Comment #18 from David Nind <david@davidnind.com> --- Created attachment 190262 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190262&action=edit Bug 40777: Add 'Audit' button in the toolbar Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #187528|0 |1 is obsolete| | --- Comment #19 from David Nind <david@davidnind.com> --- Created attachment 190263 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190263&action=edit Bug 40777: Add info in the alert Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188222|0 |1 is obsolete| | --- Comment #20 from David Nind <david@davidnind.com> --- Created attachment 190264 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190264&action=edit Bug 40777: Move all biblio tests to the new package Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188223|0 |1 is obsolete| | --- Comment #21 from David Nind <david@davidnind.com> --- Created attachment 190265 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190265&action=edit Bug 40777: Add tests for invalid_item_library and no_item_type Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188224|0 |1 is obsolete| | --- Comment #22 from David Nind <david@davidnind.com> --- Created attachment 190266 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190266&action=edit Bug 40777: Add tests for invalid_item_type Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188225|0 |1 is obsolete| | --- Comment #23 from David Nind <david@davidnind.com> --- Created attachment 190267 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190267&action=edit Bug 40777: Add tests for errors_in_marc Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188226|0 |1 is obsolete| | --- Comment #24 from David Nind <david@davidnind.com> --- Created attachment 190268 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190268&action=edit Bug 40777: Add tests for nonexistent_AV Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188227|0 |1 is obsolete| | --- Comment #25 from David Nind <david@davidnind.com> --- Created attachment 190269 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190269&action=edit Bug 40777: Add tests for empty_title Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #189598|0 |1 is obsolete| | --- Comment #26 from David Nind <david@davidnind.com> --- Created attachment 190270 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190270&action=edit Bug 40777: Flatten arrays Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Text to go in the| |This adds an "Audit" option release notes| |on the toolbar for records | |in the staff interface. | |This implements the check | |for missing home and | |current library data (952$a | |and 952$b fields). | | | |If a record has data | |inconsistencies, then using | |the audit option is shown | |on the error message that | |is shown when accessing the | |record details page: | | | | Something went | |wrong when loading the | |table. | | 500: Internal | |server error | | Have a look at the | |"Audit" button in the | |toolbar --- Comment #27 from David Nind <david@davidnind.com> --- Testing notes (using KTD): 1. Export a record with an item from the staff interface as MARCXML (Save > MARCXML). 2. Edit the file and remove the 952$a (home library) and 952$b (current library). 3. Delete the item and record in the staff interface. 4. Import the exported record and item from the MARCXML file: - Cataloging > Import > Stage for import . Settings > Format: MARCXML - Stage for import - View batch link on the job details page (assuming successfully staged) . New record framework: change to Books, Booklets, Workbooks 5. Turn on the browser developer tools and view the console. 6. Access the details page for the new record. 7. Note that you get: - A "Processing" dialog box on screen - A browser pop-up window/dialog box with: Something went wrong when loading the table. 500: Internal server error - Click OK - The console shows: . Failed to load resource: the server responded with a status of 500 (Internal Server Error) . Got 500 (Internal Server Error) . DataTables warning: table id=holdings_table - Ajax error. For more information about this error, please see https://datatables.net/tn/7 8. Run misc/maintenance/search_for_data_inconsistencies.pl, you get these result (the itemnumber may be different): == Not defined items.homebranch and/or items.holdingbranch == * Item with itemnumber=973 does not have home and holding library defined => Edit these items and set valid homebranch and/or holdingbranch 9. Apply the patch. 10. Restart everything. 11. Refresh the details screen for the imported record. 12. You now get the same as step 7, but with added "Have a look at the "Audit" button in the toolbar" text in the browser pop-up dialog box: - A "Processing" dialog box on screen - A browser pop-up window/dialog box with: Something went wrong when loading the table. 500: Internal server error Have a look at the "Audit" button in the toolbar - Click OK - The console shows: . GET http[URL for the record....] 500 (Internal Server Error) . Got 500 (Internal Server Error) . DataTables warning: table id=holdings_table - Ajax error. For more information about this error, please see https://datatables.net/tn/7 13. Click on the Audit button. 14. You get a yellow box at the top of the record with: Errors found Item with itemnumber=973 does not have home and holding library defined (Note that there is still the "Processing" dialog box, you can edit the item by select the "Items" option in the sidebar menu and then edit the item details - the home and current library files will be auto-populated with the current library, Centerville with KTD) 15. Repeat step 8, you get the same results. 16. The tests pass: prove t/db_dependent/Koha/Database/DataInconsistency.t 17. Check the details page for another record that doesn't have any data issues - note the 'Audit' toolbar item, and if clicked should show a blue box at the top of the record with "No data inconsistency found." -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |victor@tuxayo.net --- Comment #28 from Victor Grousset/tuxayo <victor@tuxayo.net> ---
move the code from the search_for_inconsistencies.pl maintenance script to a module to make it reusable from the Koha UI.
This is great :D ---- Turns out the tests fail if one happens to have started KTD with UNIMARC data. And if I go to the UI and click audit. Like http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1271 (unimarc test data) I get: The Framework ** is using the authorised value's category *TYPEDOC*, but the following biblioitems.itemtype do not have a value defined ({itemnumber => value }): {2275 => LIVR} The Framework ** is using the authorised value's category *LOC*, but the following items.location do not have a value defined ({itemnumber => value }): {2275 => Documentaires adulte} So it doesn't seems like it's just the tests that have hard coded marc21 data. ------ No POD for subs and the new module. Though the QA script skips it. Does that mean it's not actually mandated there? Or is that just because it's too hard given the legacy code everywhere to make this blocking in the script? [SKIP] Koha/Database/DataInconsistency.pm SKIP pod_coverage No POD exists ---- for_biblio() isn't covered by tests (I mean actual code coverage, not the weird error message we get in some cases) In errors_in_marc() the cases "biblio_tag < 10" and "biblioitem_tag < 10" aren't covered but if it's not trivial I don't think it's a big deal. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #29 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 190884 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190884&action=edit Bug 40777: Add tests for for_biblio Patch from commit 6fbb2b1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #30 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 190885 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190885&action=edit Bug 40777: Add POD Patch from commit 8b8f7de -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #31 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Victor Grousset/tuxayo from comment #28)
Turns out the tests fail if one happens to have started KTD with UNIMARC data.
And if I go to the UI and click audit. Like http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1271 (unimarc test data)
I get:
The Framework ** is using the authorised value's category *TYPEDOC*, but the following biblioitems.itemtype do not have a value defined ({itemnumber => value }): {2275 => LIVR} The Framework ** is using the authorised value's category *LOC*, but the following items.location do not have a value defined ({itemnumber => value }): {2275 => Documentaires adulte}
So it doesn't seems like it's just the tests that have hard coded marc21 data.
Looks like incorrect data, it's a misc4dev's problem.
No POD for subs and the new module. Though the QA script skips it. Does that mean it's not actually mandated there? Or is that just because it's too hard given the legacy code everywhere to make this blocking in the script? [SKIP] Koha/Database/DataInconsistency.pm SKIP pod_coverage No POD exists
Fixed in the last patch.
for_biblio() isn't covered by tests (I mean actual code coverage, not the weird error message we get in some cases)
Added a test.
In errors_in_marc() the cases "biblio_tag < 10" and "biblioitem_tag < 10" aren't covered but if it's not trivial I don't think it's a big deal.
Yeah, not a big deal :D And feeling lazy for that right now. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #32 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Jonathan Druart from comment #31)
No POD for subs and the new module. Fixed in the last patch.
Thanks ^^
for_biblio() isn't covered by tests (I mean actual code coverage, not the weird error message we get in some cases)
Added a test.
Coverage good! :)
Turns out the tests fail if one happens to have started KTD with UNIMARC data.
And if I go to the UI and click audit. [...] Looks like incorrect data, it's a misc4dev's problem.
Indeed, I can just create the missing values in the UI. But looking at the full test failures, and even when I create the missing values in TYPEDOC and LOC, there is also this: # Failed test at t/db_dependent/Koha/Database/DataInconsistency.t line 309. # Structures begin differing at: # $got->[0] = 'Biblionumber 21655 has '@' in 001$@' # $expected->[0] = 'Biblionumber 21655 has '-42' in 001$@' # Failed test at t/db_dependent/Koha/Database/DataInconsistency.t line 324. # Structures begin differing at: # $got->[0] = 'Biblionumber 21655 has '@' in 001$@' # $expected->[0] = 'Biblionumber 21655 has '-42' in 001$@' # Looks like you failed 2 tests of 2. # Failed test 'ids_not_in_marc' and # Failed test 'no warnings' # at /usr/share/perl/5.36/Test/Builder.pm line 193. # There were 3 warning(s) # Previous test 0 '' # Argument "@" isn't numeric in numeric ne (!=) at /kohadevbox/koha/Koha/Database/DataInconsistency.pm line 235. # at /kohadevbox/koha/Koha/Database/DataInconsistency.pm line 235. -------- There is something else weird. Still on UNIMARC, 099$t is by default TYPEDOC, which has no value. So the audit complains it can't find LIVR for example (many record have this value). - Now if I change the framework to map to itemtype. - misc/batchRebuildBiblioTables.pl - restart services - the audit will stop complaining about LIVR not existing - but it still doesn't existing in itemtype - (same result if I apply bug 38336 and start KTD) Do the steps make sense? If so, that looks like the audit let the issue slip through. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #33 from Jonathan Druart <jonathan.druart@gmail.com> --- I don't know if they make sense, but the tests are not supposed to pass with UNIMARC and I am not going to investigate why they fail. Re TYPEDOC and LIVR the behaviour with this patch should be the same as before. Do you see a change in the behaviour or was it the same on main? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190261|0 |1 is obsolete| | --- Comment #34 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191603 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191603&action=edit Bug 40777: Introduce Koha::Database::DataInconsistency test plan: 1. Export a record with an item from the staff interface as MARCXML (Save > MARCXML). 2. Edit the file and remove the 952$a (home library) and 952$b (current library). 3. Delete the item and record in the staff interface. 4. Import the exported record and item from the MARCXML file: - Cataloging > Import > Stage for import . Settings > Format: MARCXML - Stage for import - View batch link on the job details page (assuming successfully staged) . New record framework: change to Books, Booklets, Workbooks 5. Turn on the browser developer tools and view the console. 6. Access the details page for the new record. 7. Note that you get: - A "Processing" dialog box on screen - A browser pop-up window/dialog box with: Something went wrong when loading the table. 500: Internal server error - Click OK - The console shows: . Failed to load resource: the server responded with a status of 500 (Internal Server Error) . Got 500 (Internal Server Error) . DataTables warning: table id=holdings_table - Ajax error. For more information about this error, please see https://datatables.net/tn/7 8. Run misc/maintenance/search_for_data_inconsistencies.pl, you get these result (the itemnumber may be different): == Not defined items.homebranch and/or items.holdingbranch == * Item with itemnumber=973 does not have home and holding library defined => Edit these items and set valid homebranch and/or holdingbranch (↑↑ don't do it, it's part of the message content) 9. Apply the patch. 10. Restart everything. 11. Refresh the details screen for the imported record. 12. You now get the same as step 7, but with added "Have a look at the "Audit" button in the toolbar" text in the browser pop-up dialog box: - A "Processing" dialog box on screen - A browser pop-up window/dialog box with: Something went wrong when loading the table. 500: Internal server error Have a look at the "Audit" button in the toolbar - Click OK - The console shows: . GET http[URL for the record....] 500 (Internal Server Error) . Got 500 (Internal Server Error) . DataTables warning: table id=holdings_table - Ajax error. For more information about this error, please see https://datatables.net/tn/7 13. Click on the Audit button. 14. You get a yellow box at the top of the record with: Errors found Item with itemnumber=973 does not have home and holding library defined (Note that there is still the "Processing" dialog box, you can edit the item by select the "Items" option in the sidebar menu and then edit the item details - the home and current library files will be auto-populated with the current library, Centerville with KTD) 15. Repeat step 8, you get the same results. 16. The tests pass: prove t/db_dependent/Koha/Database/DataInconsistency.t 17. Check the details page for another record that doesn't have any data issues - note the 'Audit' toolbar item, and if clicked should show a blue box at the top of the record with "No data inconsistency found." Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190262|0 |1 is obsolete| | --- Comment #35 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191604 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191604&action=edit Bug 40777: Add 'Audit' button in the toolbar Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190263|0 |1 is obsolete| | --- Comment #36 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191605 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191605&action=edit Bug 40777: Add info in the alert Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190264|0 |1 is obsolete| | --- Comment #37 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191606&action=edit Bug 40777: Move all biblio tests to the new package Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190265|0 |1 is obsolete| | --- Comment #38 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191607 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191607&action=edit Bug 40777: Add tests for invalid_item_library and no_item_type Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190266|0 |1 is obsolete| | --- Comment #39 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191608 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191608&action=edit Bug 40777: Add tests for invalid_item_type Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190267|0 |1 is obsolete| | --- Comment #40 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191609 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191609&action=edit Bug 40777: Add tests for errors_in_marc Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190268|0 |1 is obsolete| | --- Comment #41 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191610 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191610&action=edit Bug 40777: Add tests for nonexistent_AV Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190269|0 |1 is obsolete| | --- Comment #42 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191611 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191611&action=edit Bug 40777: Add tests for empty_title Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190270|0 |1 is obsolete| | --- Comment #43 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191612 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191612&action=edit Bug 40777: Flatten arrays Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190884|0 |1 is obsolete| | --- Comment #44 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191613 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191613&action=edit Bug 40777: Add tests for for_biblio Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190885|0 |1 is obsolete| | --- Comment #45 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191614 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191614&action=edit Bug 40777: Add POD Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #46 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 191615 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191615&action=edit Bug 40777: (QA follow-up): remove trailing space from translatable string Otherwise it ends up as "Audit " in the .po -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #47 from Victor Grousset/tuxayo <victor@tuxayo.net> ---
the tests are not supposed to pass with UNIMARC
No worries about the tests then. I looked at the failure and it seems it's the test code that somehow puts a @. The business code gets it back so it does it's job even in UNIMARC.
Re TYPEDOC and LIVR the behavior with this patch should be the same as before. Do you see a change in the behavior or was it the same on main?
From what I understand from above, I got the audit to pass even though my record 099$t value doesn't exist in itemtypes table. I can't compare vs main, it's the new audit. Wait! it's not new, it's originally from search_for_data_inconsistencies.pl
Ok it's not from this patch. search_for_data_inconsistencies.pl can also be fooled. Including in MARC21 if I just change in the framework the authorized values from itemtypes to TYPEDOC (or any auth value category) which have not any values in MARC21 sample data. Having the same bugs is actually a good sign the extraction of business logic didn't mess things up ^^ at least in the scope of my whack tests. ---- Thanks David for the test plan :) ---- Works, makes sense, QA script happy, code looks good, passing QA :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #48 from Victor Grousset/tuxayo <victor@tuxayo.net> --- QA note: the strings are in Koha/Database/DataInconsistency.pm but I don't see an obvious way without duplicating them between the cron the and web UI. Feature still works with just "catalogue" permission. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #49 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 191687 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191687&action=edit Screenshot of proposed new menu item (In reply to Victor Grousset/tuxayo from comment #48)
Feature still works with just "catalogue" permission.
My instinct is to say that it should not have a prominent position in the toolbar and we should limit the permissions to anyone who will see the "Edit" dropdown: CAN_user_editcatalogue_edit_catalogue || CAN_user_editcatalogue_edit_items || CAN_user_tools_items_batchmod || CAN_user_tools_items_batchdel ) or ( frameworkcode == 'FA' and CAN_user_editcatalogue_fast_cataloging -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |victor@tuxayo.net |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |26.05.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #50 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to main for 26.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #51 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Quite a few failing Cypress tests here now: SyntaxError: The following error originated from your application code, not from Cypress.
Identifier 'specific_dt_errors' has already been declared
specific_dt_errors already declared either means we are including the datatables.js asset twice? Or it is a JS scoping issue with let vs var? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #52 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Created attachment 191821 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191821&action=edit Bug 40777: (follow-up) Use var instead of let when declaring specific_dt_errors -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | --- Comment #53 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- follow-up pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #191821|0 |1 is obsolete| | --- Comment #54 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 191835 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191835&action=edit Bug 40777: (follow-up) Use var instead of let when declaring specific_dt_errors Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #55 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Lucas Gass (lukeg) from comment #51)
Quite a few failing Cypress tests here now:
SyntaxError: The following error originated from your application code, not from Cypress.
Identifier 'specific_dt_errors' has already been declared
specific_dt_errors already declared either means we are including the datatables.js asset twice? Or it is a JS scoping issue with let vs var?
Yes, the fix is correct, Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #56 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Owen Leonard from comment #49)
My instinct is to say that it should not have a prominent position in the toolbar and we should limit the permissions to anyone who will see the "Edit" dropdown:
CAN_user_editcatalogue_edit_catalogue || CAN_user_editcatalogue_edit_items || CAN_user_tools_items_batchmod || CAN_user_tools_items_batchdel ) or ( frameworkcode == 'FA' and CAN_user_editcatalogue_fast_cataloging
Good point. I was thinking it's ok to have everyone see the button to be able to more helpfully report an eventual issue to their admin or a cataloger. But I'm not sure if that is that much valuable. So the button is indeed too prominent just for that. That would definitely be the next work increment on that. Thanks for the snippet with the permission checks. That will be useful if someone else can continue to work on this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Jacob O'Mara <jacob.omara@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Pushed to stable Version(s)|26.05.00 |26.05.00,25.11.02 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #57 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Thanks all, pushed to 25.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |baptiste.wojtkowski@biblibr | |e.com --- Comment #58 from Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> --- This patch appears to slow the inconsistencies checks on big databases (from about 15-30 mins to 1 day on some customers). Calls like the following one result in a SQL IN that is breaking performances.
my $items_with_invalid_itype = Koha::Items->search( { biblionumber => $ids, -and => [ itype => { not_in => \@itemtypes }, itype => { '!=' => '' > } ] } );
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42846 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42846 [Bug 42846] Koha::Database::DataInconsistency typo for biblioitemnumber in MARC -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com Documentation| |Caroline Cyr La Rose contact| | Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/1206 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40777 --- Comment #59 from Fridolin Somers <fridolin.somers@biblibre.com> --- 25.05.x RMaint : This is nearly enhancement so I propose not to backport on 25.05.x -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org