[Bug 34029] New: Import breaks on too long publishercode
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Bug ID: 34029 Summary: Import breaks on too long publishercode Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Cataloging Assignee: koha-bugs@lists.koha-community.org Reporter: m.de.rooy@rijksmuseum.nl QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Data too long for column 'publishercode' at row 1 at /usr/share/koha/Koha/Object.pm line 170 DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Data too long for column 'publishercode' at row 1 at /usr/share/koha/Koha/Object.pm line 170 Use of uninitialized value $record_number in concatenation (.) or string at /usr/share/koha/C4/Biblio.pm line 2431. C4::Biblio::ModZebra(): DBI Exception: DBD::mysql::st execute failed: Column 'biblio_auth_number' cannot be null at /usr/share/koha/Koha/SearchEngine/Zebra/Indexer.pm line 61 ModZebra( $record_number, $op, $server ); Whats happening here? Two records have a too long publishercode. Two undef's (from AddBiblio) are pushed to array @biblio_ids. Indexing crashes after the import. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #1 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- <marcelr> we should prevent the indexing crash very simple by testing what we push to biblio_ids -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #2 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- We could extend publishercode to TEXT -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Bug 28328 increased several columns to longtext, biblio is almost completely longtext, so I suggest to do the same for the remaining varchars in biblioitems: * place * publishercode * illus * pages * size -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=28328 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |katrin.fischer@bsz-bw.de |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Import breaks on too long |Import breaks when data |publishercode |exceeds size of mapped | |database columns -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 152404 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152404&action=edit Bug 34029: Extend datatyps of common biblioitems.columns to longtext to avoid breaking import This updated the datatype of the following columns from varchar(255) to longtext to avoid import problems if the mapped MARC field's data exceeds 255 characters. This is for: * place (260$a and 264$a) * publishercode (260$b and 264$b) * size (300$c) * illus (300$b) * pages (300$a) To test: * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. Try to save. * Run the updatedatabase script to modify the DB structure * restart_all * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. * Save * Export the record * Import the record using staging tools, all should work well -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jan.kissig@th-wildau.de, | |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #152404|0 |1 is obsolete| | --- Comment #5 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 152417 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152417&action=edit Bug 34029: Extend datatyps of common biblioitems.columns to longtext to avoid breaking import This updated the datatype of the following columns from varchar(255) to longtext to avoid import problems if the mapped MARC field's data exceeds 255 characters. This is for: * place (260$a and 264$a) * publishercode (260$b and 264$b) * size (300$c) * illus (300$b) * pages (300$a) To test: * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. Try to save. * Run the updatedatabase script to modify the DB structure * restart_all * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. * Save * Export the record * Import the record using staging tools, all should work well -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #152417|0 |1 is obsolete| | --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 152418 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152418&action=edit Bug 34029: Extend datatypes of biblioitems.columns to longtext to avoid breaking import This updated the datatype of the following columns from varchar(255) to longtext to avoid import problems if the mapped MARC field's data exceeds 255 characters. This is for: * place (260$a and 264$a) * publishercode (260$b and 264$b) * size (300$c) * illus (300$b) * pages (300$a) To test: * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. Try to save. * Run the updatedatabase script to modify the DB structure * restart_all * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. * Save * Export the record * Import the record using staging tools, all should work well -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #7 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Katrin Fischer from comment #3)
Bug 28328 increased several columns to longtext, biblio is almost completely longtext, so I suggest to do the same for the remaining varchars in biblioitems:
* place * publishercode * illus * pages * size
Might not really matter, but LONGTEXT (4G) seems to be absolute overkill for these five fields. Tinytext might be just too small. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Marcel de Rooy from comment #7)
(In reply to Katrin Fischer from comment #3)
Bug 28328 increased several columns to longtext, biblio is almost completely longtext, so I suggest to do the same for the remaining varchars in biblioitems:
* place * publishercode * illus * pages * size
Might not really matter, but LONGTEXT (4G) seems to be absolute overkill for these five fields. Tinytext might be just too small.
Mostly picked longtext because the majority of datatypes is already longtext if you look at biblio and biblioitems. What do you suggest? We also have a fair bit of mediumtext. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #9 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Katrin Fischer from comment #8)
(In reply to Marcel de Rooy from comment #7)
(In reply to Katrin Fischer from comment #3)
Bug 28328 increased several columns to longtext, biblio is almost completely longtext, so I suggest to do the same for the remaining varchars in biblioitems:
* place * publishercode * illus * pages * size
Might not really matter, but LONGTEXT (4G) seems to be absolute overkill for these five fields. Tinytext might be just too small.
Mostly picked longtext because the majority of datatypes is already longtext if you look at biblio and biblioitems. What do you suggest? We also have a fair bit of mediumtext.
Comment2 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Sam Lau <samalau@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Sam Lau <samalau@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #152418|0 |1 is obsolete| | --- Comment #10 from Sam Lau <samalau@gmail.com> --- Created attachment 152490 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152490&action=edit Bug 34029: Extend datatypes of biblioitems.columns to longtext to avoid breaking import This updated the datatype of the following columns from varchar(255) to longtext to avoid import problems if the mapped MARC field's data exceeds 255 characters. This is for: * place (260$a and 264$a) * publishercode (260$b and 264$b) * size (300$c) * illus (300$b) * pages (300$a) To test: * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. Try to save. * Run the updatedatabase script to modify the DB structure * restart_all * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. * Save * Export the record * Import the record using staging tools, all should work well Signed-off-by: Sam Lau <samalau@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #11 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Please change to TEXT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #152490|0 |1 is obsolete| | --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 152647 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152647&action=edit Bug 34029: Extend datatypes of biblioitems.columns to longtext to avoid breaking import This updated the datatype of the following columns from varchar(255) to longtext to avoid import problems if the mapped MARC field's data exceeds 255 characters. This is for: * place (260$a and 264$a) * publishercode (260$b and 264$b) * size (300$c) * illus (300$b) * pages (300$a) To test: * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. Try to save. * Run the updatedatabase script to modify the DB structure * restart_all * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. * Save * Export the record * Import the record using staging tools, all should work well -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 152648 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152648&action=edit Bug 34029: (QA follow-up) Change from longtext to text Can be squashed if needed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Marcel de Rooy from comment #11)
Please change to TEXT
I've made the requested change - in a follow-up for easier review. Please feel free to squash later. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #152647|0 |1 is obsolete| | Attachment #152648|0 |1 is obsolete| | --- Comment #15 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 152670 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152670&action=edit Bug 34029: Extend datatypes of biblioitems.columns to text to avoid breaking import This updated the datatype of the following columns from varchar(255) to text to avoid import problems if the mapped MARC field's data exceeds 255 characters. This is for: * place (260$a and 264$a) * publishercode (260$b and 264$b) * size (300$c) * illus (300$b) * pages (300$a) To test: * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. Try to save. * Run the updatedatabase script to modify the DB structure * restart_all * Edit a bibliographic record and enter more than 255 characters into the MARC fields listed above. * Save * Export the record * Import the record using staging tools, all should work well Signed-off-by: Sam Lau <samalau@gmail.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #16 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- RM: Please add DBIC changes too. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #17 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 152671 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152671&action=edit Bug 34029: (QA follow-up) Fix pushing undef to biblio_ids See comment1. Although we now fix the error on publishercode, it is good to verify the result before pushing. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |23.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #18 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #19 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Comment on attachment 152670 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152670 Bug 34029: Extend datatypes of biblioitems.columns to text to avoid breaking import Marcel:
+ `place` text DEFAULT NULL COMMENT 'publication place (MARC21 260$a and 264§a)',
Katrin's patch didn't contain that char. You are evil :-D -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #20 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 152866 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=152866&action=edit Bug 34029: (QA follow-up) Fix slipped broken char Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #21 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Tomás Cohen Arazi from comment #20)
Created attachment 152866 [details] [review] Bug 34029: (QA follow-up) Fix slipped broken char
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Good catch :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #22 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think it was me after all, thx for catching it! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.11.00 |23.11.00,23.05.02 released in| | Status|Pushed to master |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #23 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Thanks for all the hard work! Pushed to 23.05.x for the next release -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Pedro Amorim <pedro.amorim@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.11.00,23.05.02 |23.11.00,23.05.02,22.11.08 released in| | Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #24 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- Nice work everyone! Pushed to 22.11.x for next release -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #25 from Fridolin Somers <fridolin.somers@biblibre.com> --- There is a small typo in kohastructure.sql : `publishercode` text DEFAULT NULL COMMENT 'publisher (MARC21 260$b and 246$b)', 246$b => 264$b Maybe just fix in master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |34548 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34548 [Bug 34548] Upgrade failed on conversion of publishercode from varchar to text on biblioitems and deletedbiblioitems due to existing indexes -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |34516 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34516 [Bug 34516] Upgrade database fails for 22.11.07.003, points to web installer -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35180 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35180 [Bug 35180] Fix typo in deletedbiblioitems.publishercode comment in kohastructure.sql -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 --- Comment #26 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Fridolin Somers from comment #25)
There is a small typo in kohastructure.sql : `publishercode` text DEFAULT NULL COMMENT 'publisher (MARC21 260$b and 246$b)',
246$b => 264$b
Maybe just fix in master
As this has already been backported, I've opened a separate bug: Bug 35180 - Fix typo in deletedbiblioitems.publishercode comment in kohastructure.sql -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |36978 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36978 [Bug 36978] Issue upgrading to 24.05 from 22.05 - publishercode -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34029 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |wainuiwitikapark@catalyst.n | |et.nz Status|Pushed to oldstable |RESOLVED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org