[Bug 26853] New: "Data too long for column" errors during MARC import
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Bug ID: 26853 Summary: "Data too long for column" errors during MARC import Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: MARC Bibliographic record staging/import Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org I'm noticing "Data too long for column 'title'" and "Data too long for column 'issn'" database errors in the logs. These are preventing import_biblios rows from being added to the system, although import_records rows are still being added. On koha-testing-docker, these errors are being silently ignored, which is not good. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- One option is to try to match the import_biblios columns with the biblio and biblioitems tables... I notice import_biblios has an index on title, so I don't know if that will truncate the title for the index, or if it will just fail. If people don't clean their import_biblios table, this could also be a massive database update later, which would not be fun... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Another option would be to truncate the data before trying to insert it into the database. But considering that things like Koha::MetaSearcher use those fields... maybe we should fix them. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Created attachment 112656 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112656&action=edit Bug 26853: Throw a fatal error if import_biblios insert fails -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Created attachment 112657 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112657&action=edit Bug 26853: Harmonize import_biblios with biblio and biblioitems This patch harmonizes the column datatypes of import_biblios with biblio and biblioitems to prevent database errors which cause staged MARC uploads to sometimes fail to update 100% correctly. To test: 1. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 2. Upload a file with MARC records with titles longer than 128 characters long 3. Click "Stage for import" 4. Monitor /var/log/koha/kohadev/intranet-error.log for database errors 5. Note that there are no database errors -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #112656|0 |1 is obsolete| | --- Comment #5 from David Cook <dcook@prosentient.com.au> --- Created attachment 112658 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112658&action=edit Bug 26853: Throw a fatal error if import_biblios insert fails -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | Summary|"Data too long for column" |Data lost due to "Data too |errors during MARC import |long for column" errors | |during MARC import Severity|minor |major -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|major |critical -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #6 from David Nind <david@davidnind.com> --- Hi David. Are you able to provide a sample file for import with records that will trigger this? I tried by editing a record and making the title longer, then exporting; but it still worked. Otherwise, I will have another go this evening. David -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- (In reply to David Nind from comment #6)
Hi David.
Are you able to provide a sample file for import with records that will trigger this?
I tried by editing a record and making the title longer, then exporting; but it still worked.
Otherwise, I will have another go this evening.
David
Were you testing the patch or trying to reproduce the error? The problem is really difficult to spot, but I'll look at putting together a sample set of data. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Created attachment 112884 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112884&action=edit Record with long title -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- I've added a record with a long title. Without the patch, when I go to view the import batch (at /cgi-bin/koha/tools/manage-marc-import.pl?import_batch_id=1), I see that the "Citation" says "null". If I go into the database, I can see 1 record in import_records, but there are 0 records in import_biblios. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #112657|0 |1 is obsolete| | --- Comment #10 from David Nind <david@davidnind.com> --- Created attachment 112885 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112885&action=edit Bug 26853: Harmonize import_biblios with biblio and biblioitems This patch harmonizes the column datatypes of import_biblios with biblio and biblioitems to prevent database errors which cause staged MARC uploads to sometimes fail to update 100% correctly. To test: 1. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 2. Upload a file with MARC records with titles longer than 128 characters long 3. Click "Stage for import" 4. Monitor /var/log/koha/kohadev/intranet-error.log for database errors 5. Note that there are no database errors Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #112658|0 |1 is obsolete| | --- Comment #11 from David Nind <david@davidnind.com> --- Created attachment 112886 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112886&action=edit Bug 26853: Throw a fatal error if import_biblios insert fails Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #12 from David Nind <david@davidnind.com> --- Thanks David! I had tried to recreate the error first, but I had no luck. The record you provided really helped! I hadn't spotted that a database was required as well. Everything appears to work now for me so I have signed off. I did notice another error in the database - it happened both before and after the patch was applied, but I don't think this is related to this bug: [Tue Nov 03 02:57:55.761624 2020] [cgi:error] [pid 3980] [client 172.18.0.1: 58304] AH01215: [Tue Nov 3 02:57:55 2020] stage-marc-import.pl: Filehandle STDOUT reopened as $fh only for input at /usr/share/perl5/DateTime/TimeZone/ Local/Unix.pm line 146.: /kohadevbox/koha/tools/stage-marc-import.pl, refere r: http://127.0.0.1:8081/cgi-bin/koha/tools/stage-marc-import.pl David -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- (In reply to David Nind from comment #12)
Everything appears to work now for me so I have signed off.
Thanks, David. Much appreciated! I think is a really nasty bug that is easy to miss, especially when RaiseError is turned off. When Bug 25026 is pushed, it will make issues like this more obvious.
I did notice another error in the database - it happened both before and after the patch was applied, but I don't think this is related to this bug:
[Tue Nov 03 02:57:55.761624 2020] [cgi:error] [pid 3980] [client 172.18.0.1: 58304] AH01215: [Tue Nov 3 02:57:55 2020] stage-marc-import.pl: Filehandle STDOUT reopened as $fh only for input at /usr/share/perl5/DateTime/TimeZone/ Local/Unix.pm line 146.: /kohadevbox/koha/tools/stage-marc-import.pl, refere r: http://127.0.0.1:8081/cgi-bin/koha/tools/stage-marc-import.pl
That's more of a warning than an error. It's not related to this bug, but I have been thinking of looking into it more. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #112885|0 |1 is obsolete| | --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 112888 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112888&action=edit Bug 26853: Harmonize import_biblios with biblio and biblioitems This patch harmonizes the column datatypes of import_biblios with biblio and biblioitems to prevent database errors which cause staged MARC uploads to sometimes fail to update 100% correctly. To test: 1. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 2. Upload a file with MARC records with titles longer than 128 characters long 3. Click "Stage for import" 4. Monitor /var/log/koha/kohadev/intranet-error.log for database errors 5. Note that there are no database errors Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #112886|0 |1 is obsolete| | --- Comment #15 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 112889 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112889&action=edit Bug 26853: Throw a fatal error if import_biblios insert fails Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |martin.renvoize@ptfs-europe | |.com --- Comment #16 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- It makes sense for these tables to be harmonized and I'm happy with the error being thrown on failures. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Pushed to master |Pushed to stable Version(s)|20.11.00 |20.11.00, 20.05.06 released in| | --- Comment #18 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 20.05.x for 20.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz --- Comment #19 from Aleisha Amohia <aleisha@catalyst.net.nz> --- I can't reproduce the errors in the logs in 19.11.x. not backported -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 --- Comment #20 from David Cook <dcook@prosentient.com.au> --- (In reply to Aleisha Amohia from comment #19)
I can't reproduce the errors in the logs in 19.11.x. not backported
That's interesting since 19.11.something is where I discovered the bug. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26853 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #21 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- *** Bug 26521 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org