https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |17197 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17197 [Bug 17197] misc/batchupdateISBNs.pl is certainly no longer in use -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |17193 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17193 [Bug 17193] C4::Search::SearchAcquisitions is not used -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 54910 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54910&action=edit Bug 17196: Move marcxml out of the biblioitems table Two discussions on koha-devel lead to the same conclusion: biblioitems.marcxml should be moved out this table - biblio and biblioitems http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html - biblioitems.marcxml & biblioitems.marc / HUGE performance issue ! http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html There are several goals to do it: - Performance As Paul Poulain wrote, a simple query like SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear; takes more than 10min on a DB with more than 1M bibliographic records but only 3sec (!) on the same DB without the biblioitems.marcxml field Note that priori to this patch set, the biblioitems.marcxml was not retrieved systematically, but was, at least, in C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders - Flexibility Storing the marcxml in a specific table would allow use to store several kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats (marcflavour) - Clean code It would be a first step toward Koha::MetadataRecord for bibliographic records (not done in this patch set). Test plan: - Update the DBIC Schema - Add / Edit / Delete / Import / Export bibliographic records - Add items - Reindex records using ES - Confirm that the following scripts still work: * misc/cronjobs/delete_records_via_leader.pl * misc/migration_tools/build_oai_sets.pl - Look at the reading history at the OPAC (opac-readingrecord.pl) - At the OPAC, click on a tag, you must see the result Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 54912 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54912&action=edit Bug 17196: Update misc/maintenance/sanitize_records.pl Test plan: Use the sanitize_records.pl maintenance script with the --auto-search option -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 54914 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54914&action=edit Bug 17196: DB Changes Remove biblioitems.marcxml and deletedbiblioitems.marcxml and adds 2 table biblio_metadata and deletedbiblio_metadata -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Bob Birchall <bob@calyx.net.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bob@calyx.net.au -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #7 from Mason James <mtj@kohaaloha.com> --- Hi Jonathan I attempted to test this patchset, but I had a big problem trying to 'git bz apply' the 1st patch, on master (0254fd7) 0254fd7 Bug 17135 - DBRev 16.06.00.028 would you take another look at the patchset please? --------------- mason@xen1:~/g/k/Koha$ git bz apply 17196 Bug 17196 - Move marcxml out of the biblioitems table 54910 - Bug 17196: Move marcxml out of the biblioitems table 54911 - Bug 17196: Fix query builder for item search 54912 - Bug 17196: Update misc/maintenance/sanitize_records.pl 54913 - Bug 17196: Update switch_marc21_series_info.pl 54918 - Bug 17196: DB Changes Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 17196: Move marcxml out of the biblioitems table fatal: sha1 information is lacking or useless (C4/Biblio.pm). Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 17196: Move marcxml out of the biblioitems table The copy of the patch that failed is found in: /home/mason/g/k/Koha/.git/rebase-apply/patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-17196-Move-marcxml-out-of-the-biblioitems-tabl-jCnYTS.patch -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #9 from Owen Leonard <oleonard@myacpl.org> ---

...

I attempted to test this patchset, but I had a big problem trying to 'git bz apply' the 1st patch, on master (0254fd7)

git bz didn't apply this successfully when using the auto-inclusion of dependent patches, but I don't know why. Applying the patches manually works without error. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #11 from Mason James <mtj@kohaaloha.com> --- (In reply to Mason James from comment #10)

(In reply to Jonathan Druart from comment #8)

...

(In reply to Mason James from comment #7)

...

Hi Jonathan

I attempted to test this patchset, but I had a big problem trying to 'git bz apply' the 1st patch, on master (0254fd7)

0254fd7 Bug 17135 - DBRev 16.06.00.028

would you take another look at the patchset please?

Everything is applying ok for me, please check the dependencies.

afaik (In reply to Jonathan Druart from comment #8)

...

(In reply to Mason James from comment #7)

...

Hi Jonathan

I attempted to test this patchset, but I had a big problem trying to 'git bz apply' the 1st patch, on master (0254fd7)

0254fd7 Bug 17135 - DBRev 16.06.00.028

would you take another look at the patchset please?

Everything is applying ok for me, please check the dependencies.

i have all listed dependencies applied on commit 56da7be (10455 17193 17197)

the patch still fails for me

here's the transcript... mason@xen1:~/g/k/Koha$ git log --oneline' 56da7be Bug 17352 - Patron search type is hard coded to 'contain' in circ/circulation.pl 5eb4b06 Bug 16273 - DBRev 16.06.00.031 3d4dc58 Bug 16273: Add the ability not to display the login details for newly registered patrons 8d0b412 Bug 16273: Add the new pref PatronSelfRegistrationPrefillForm d85e80b Bug 17301: Follow-up - Standardize headings mason@xen1:~/g/k/Koha$ git bz apply 10455 Bug 10455 - remove redundant 'biblioitems.marc' field 55844 - Bug 10455: Remove biblioitems.marc - DB changes 55845 - Bug 10455: Remove biblioitems.marc 55846 - Bug 10455: Remove biblioitems.marc - rm MARC21_parse_test.pl 55847 - Bug 10455: Remove biblioitems.marc - update delete_records_via_leader.pl 55848 - Bug 10455: (follow-up) Remove biblioitems.marc 55849 - Bug 10455: (follow-up 2) Remove biblioitems.marc 55850 - Bug 10455: (follow-up 3) Remove biblioitems.marc 55851 - Bug 10455: Remove MARC21_utf8_flag_fix.pl 55852 - Bug 10455: Fix tests 55853 - Bug 10455 (QA Followup) Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 10455: Remove biblioitems.marc - DB changes Applying: Bug 10455: Remove biblioitems.marc Applying: Bug 10455: Remove biblioitems.marc - rm MARC21_parse_test.pl Applying: Bug 10455: Remove biblioitems.marc - update delete_records_via_leader.pl Applying: Bug 10455: (follow-up) Remove biblioitems.marc Applying: Bug 10455: (follow-up 2) Remove biblioitems.marc Applying: Bug 10455: (follow-up 3) Remove biblioitems.marc Applying: Bug 10455: Remove MARC21_utf8_flag_fix.pl Applying: Bug 10455: Fix tests Applying: Bug 10455 (QA Followup) mason@xen1:~/g/k/Koha$ mason@xen1:~/g/k/Koha$ git bz apply 17193 Bug 17193 - C4::Search::SearchAcquisitions is not used 54962 - [PASSED QA] Bug 17193: Remove C4::Search::SearchAcquisitions Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 17193: Remove C4::Search::SearchAcquisitions Using index info to reconstruct a base tree... M C4/Search.pm Falling back to patching base and 3-way merge... No changes -- Patch already applied. mason@xen1:~/g/k/Koha$ git bz apply 17197 Bug 17197 - misc/batchupdateISBNs.pl is certainly no longer in use 55478 - [PASSED QA] Bug 17197: Remove misc/batchupdateISBNs.pl Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 17197: Remove misc/batchupdateISBNs.pl Using index info to reconstruct a base tree... A misc/batchupdateISBNs.pl Falling back to patching base and 3-way merge... No changes -- Patch already applied. mason@xen1:~/g/k/Koha$ mason@xen1:~/g/k/Koha$ git log --oneline' 4f6b1ee Bug 10455 (QA Followup) 98af430 Bug 10455: Fix tests a5e6f98 Bug 10455: Remove MARC21_utf8_flag_fix.pl fbb280d Bug 10455: (follow-up 3) Remove biblioitems.marc 81a0fb2 Bug 10455: (follow-up 2) Remove biblioitems.marc 4e3520d Bug 10455: (follow-up) Remove biblioitems.marc 339d2bc Bug 10455: Remove biblioitems.marc - update delete_records_via_leader.pl d572063 Bug 10455: Remove biblioitems.marc - rm MARC21_parse_test.pl dccc79a Bug 10455: Remove biblioitems.marc 558f6b1 Bug 10455: Remove biblioitems.marc - DB changes 56da7be Bug 17352 - Patron search type is hard coded to 'contain' in circ/circulation.pl 5eb4b06 Bug 16273 - DBRev 16.06.00.031 mason@xen1:~/g/k/Koha$ git bz apply 17196 Bug 17196 - Move marcxml out of the biblioitems table 54910 - Bug 17196: Move marcxml out of the biblioitems table 54911 - Bug 17196: Fix query builder for item search 54912 - Bug 17196: Update misc/maintenance/sanitize_records.pl 54913 - Bug 17196: Update switch_marc21_series_info.pl 54918 - Bug 17196: DB Changes Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 17196: Move marcxml out of the biblioitems table fatal: sha1 information is lacking or useless (C4/Biblio.pm). Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 17196: Move marcxml out of the biblioitems table The copy of the patch that failed is found in: /home/mason/g/k/Koha/.git/rebase-apply/patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-17196-Move-marcxml-out-of-the-biblioitems-tabl-HBdRv1.patch mason@xen1:~/g/k/Koha$ git apply /tmp/Bug-17196-Move-marcxml-out-of-the-biblioitems-tabl-HBdRv1.patch error: patch failed: C4/Biblio.pm:42 error: C4/Biblio.pm: patch does not apply error: patch failed: C4/ILSDI/Services.pm:406 error: C4/ILSDI/Services.pm: patch does not apply error: patch failed: misc/cronjobs/delete_records_via_leader.pl:68 error: misc/cronjobs/delete_records_via_leader.pl: patch does not apply -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #14 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 55857 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55857&action=edit Bug 17196: Fix query builder for item search The item search needs to join on biblio_metadata to allow search on marcxml field Test plan: Launch complex item searches (using marc fields). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 55859 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55859&action=edit Bug 17196: Update switch_marc21_series_info.pl Not sure if this script is still used, could someone confirm? Test plan: If you know how to test it, please do -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #55856|0 |1 is obsolete| | --- Comment #18 from Mason James <mtj@kohaaloha.com> --- Created attachment 55904 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55904&action=edit Bug 17196: Move marcxml out of the biblioitems table Two discussions on koha-devel lead to the same conclusion: biblioitems.marcxml should be moved out this table - biblio and biblioitems http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html - biblioitems.marcxml & biblioitems.marc / HUGE performance issue ! http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html There are several goals to do it: - Performance As Paul Poulain wrote, a simple query like SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear; takes more than 10min on a DB with more than 1M bibliographic records but only 3sec (!) on the same DB without the biblioitems.marcxml field Note that priori to this patch set, the biblioitems.marcxml was not retrieved systematically, but was, at least, in C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders - Flexibility Storing the marcxml in a specific table would allow use to store several kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats (marcflavour) - Clean code It would be a first step toward Koha::MetadataRecord for bibliographic records (not done in this patch set). Test plan: - Update the DBIC Schema - Add / Edit / Delete / Import / Export bibliographic records - Add items - Reindex records using ES - Confirm that the following scripts still work: * misc/cronjobs/delete_records_via_leader.pl * misc/migration_tools/build_oai_sets.pl - Look at the reading history at the OPAC (opac-readingrecord.pl) - At the OPAC, click on a tag, you must see the result Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108. Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #55858|0 |1 is obsolete| | --- Comment #20 from Mason James <mtj@kohaaloha.com> --- Created attachment 55906 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55906&action=edit Bug 17196: Update misc/maintenance/sanitize_records.pl Test plan: Use the sanitize_records.pl maintenance script with the --auto-search option Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #55860|0 |1 is obsolete| | --- Comment #22 from Mason James <mtj@kohaaloha.com> --- Created attachment 55908 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55908&action=edit Bug 17196: DB Changes Remove biblioitems.marcxml and deletedbiblioitems.marcxml and adds 2 table biblio_metadata and deletedbiblio_metadata Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #24 from Mason James <mtj@kohaaloha.com> --- (In reply to Mason James from comment #23)

Hi Folks I have made some good progress testing this patch-set

I did hit a problem, when attempting to save a bib record via the cataloguing editor I added a "DBI->trace(2,'/tmp/some.log')" to the start of the ModBiblio() sub

When saving a bib, I noticed that there is no SQL in the log to UPDATE the 'biblio_metadata' table

a quick grep thru the dbi.log confirms this (i think?) $ grep UPDATE /tmp/some.log -> prepare for DBD::mysql::db (DBI::db=HASH(0x55203c8)~0x552f888 'UPDATE biblio SET frameworkcode=? WHERE biblionumber=?') thr#1a20010 -> dbd_st_prepare MYSQL_VERSION_ID 50550, SQL statement: UPDATE biblio SET frameworkcode=? WHERE biblionumber=?

count_params statement UPDATE biblio SET frameworkcode=? WHERE biblionumber=? parse_params statement UPDATE biblio SET frameworkcode=? WHERE biblionumber=? Binding parameters: UPDATE biblio SET frameworkcode='BKS' WHERE biblionumber='3449' UPDATE biblio UPDATE biblio UPDATE biblio UPDATE biblio Binding parameters: UPDATE biblio -> prepare for DBD::mysql::db (DBI::db=HASH(0x55203c8)~0x552f888 'UPDATE biblioitems -> dbd_st_prepare MYSQL_VERSION_ID 50550, SQL statement: UPDATE biblioitems count_params statement UPDATE biblioitems parse_params statement UPDATE biblioitems Binding parameters: UPDATE biblioitems

-- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #26 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 55921 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55921&action=edit Bug 17196: Simplify tests for the different marc flavours centralize the way MARC field are set. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #28 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Mason James from comment #25)

(In reply to Mason James from comment #23)

...

Hi Folks I have made some good progress testing this patch-set

I did hit a problem, when attempting to save a bib record via the cataloguing editor

could someone else confirm this bug, please?

Confirmed! Sorry about that, I should have written these tests before... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #55922|0 |1 is obsolete| | --- Comment #30 from Mason James <mtj@kohaaloha.com> --- Created attachment 55951 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55951&action=edit Bug 17196: Fix - Update metadata on update Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #32 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 56038 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=56038&action=edit Bug 17196: follow-up for bug 17088 If bug 17088 is pushed first, we will need this patch. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Hugo Agud <hagud@orex.es> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #35 from Hugo Agud <hagud@orex.es> --- sorry, I published the conflict in the wrong bug -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #55905|0 |1 is obsolete| | --- Comment #37 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 57836 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=57836&action=edit Bug 17196: Fix query builder for item search The item search needs to join on biblio_metadata to allow search on marcxml field Test plan: Launch complex item searches (using marc fields). Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #55907|0 |1 is obsolete| | --- Comment #39 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 57838 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=57838&action=edit Bug 17196: Update switch_marc21_series_info.pl Not sure if this script is still used, could someone confirm? Test plan: If you know how to test it, please do Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #55950|0 |1 is obsolete| | --- Comment #41 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 57840 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=57840&action=edit Bug 17196: Simplify tests for the different marc flavours centralize the way MARC field are set. Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #56038|0 |1 is obsolete| | --- Comment #43 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 57842 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=57842&action=edit Bug 17196: follow-up for bug 17088 If bug 17088 is pushed first, we will need this patch. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.tajoli@cineca.it See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=17088 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #57842|0 |1 is obsolete| | --- Comment #46 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 58000 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58000&action=edit Bug 17196: follow-up for bug 17088 If bug 17088 is pushed first, we will need this patch. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com --- Comment #48 from Josef Moravec <josef.moravec@gmail.com> --- So, I started to test this, it looks good so far only few thing: 1. build_oai_sets.pl script does return this error on every biblio: (biblio 3) Error while creating record from marcxml: Empty String at /usr/share/perl5/MARC/File/XML.pm line 448. 2. When I click on tag in opac, I get ISE, the log says: Can't locate object method "subfield" via package "<?xml version="1.0" encoding="UTF-8"?><record .. whole xml record" at C4/Search.pm line 1925 3. The usage text for --auto-search paramater in sanitize_records.pl still says: "Automatically search records containing "&" in biblioitems.marcxml or in the specified fields." -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #50 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Martin Renvoize from comment #49)

(In reply to Martin Renvoize from comment #33)

No response to this :(

Personally, I'd use JSON as both Postgres (https://www.postgresql.org/docs/current/static/datatype-json.html) and MySQL (https://dev.mysql.com/doc/refman/5.7/en/json.html) have pretty extensive support for marked performance gains using JSON typed fields.

The first goal of these patches is to move the data out of this table to bring speed improvement. Afterwards we will think about what we can do with this new table :) As you know, we are not ready yet to support another DBMS. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #52 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 58008 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58008&action=edit Bug 17196: Update POD of sanitise_records.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #54 from Zeno Tajoli <z.tajoli@cineca.it> --- Hi Jonathan, I think there problems on "Fix query builder for item search". I try to do a search with a personalized index based on 245$c. The search doesn't use the index based on 245$c. But it use the standard indexes (based on table 'items'). No errors in intranet logs Bye -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #56 from Zeno Tajoli <z.tajoli@cineca.it> --- Hi, I go here:Home › Administration › Item search fields and I add a index based on tag 245$c. But I check the default here: http://kohastaffsandbox.ptfs-europe.co.uk and also on master it doesn't work. So it is an other bug. Bye -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #58 from Josef Moravec <josef.moravec@gmail.com> --- (In reply to Jonathan Druart from comment #53)

(In reply to Josef Moravec from comment #48)

...

2. When I click on tag in opac, I get ISE, the log says: Can't locate object method "subfield" via package "<?xml version="1.0" encoding="UTF-8"?><record .. whole xml record" at C4/Search.pm line 1925

It's a bug from master, please open a new bug report. Note that it only happens when SearchEngine=Elastic

Done in bug 17739 as you have seen ;) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #60 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 58024 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58024&action=edit [SIGNED-OFF] Bug 17196: Fix query builder for item search The item search needs to join on biblio_metadata to allow search on marcxml field Test plan: Launch complex item searches (using marc fields). Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #62 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 58026 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58026&action=edit [SIGNED-OFF] Bug 17196: Update switch_marc21_series_info.pl Not sure if this script is still used, could someone confirm? Test plan: If you know how to test it, please do Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #64 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 58028 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58028&action=edit [SIGNED-OFF] Bug 17196: Simplify tests for the different marc flavours centralize the way MARC field are set. Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #66 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 58030 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58030&action=edit [SIGNED-OFF] Bug 17196: follow-up for bug 17088 If bug 17088 is pushed first, we will need this patch. Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58023|0 |1 is obsolete| | --- Comment #69 from Zeno Tajoli <z.tajoli@cineca.it> --- Created attachment 58037 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58037&action=edit Move marcxml out of the biblioitems table -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #71 from Zeno Tajoli <z.tajoli@cineca.it> --- Created attachment 58039 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58039&action=edit [SIGNED-OFF] Bug 17196: Move marcxml out of the biblioitems table -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #73 from Zeno Tajoli <z.tajoli@cineca.it> --- Created attachment 58041 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58041&action=edit [SIGNED-OFF] Bug 17196: Update misc/maintenance/sanitize_records.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58027|0 |1 is obsolete| | --- Comment #75 from Zeno Tajoli <z.tajoli@cineca.it> --- Created attachment 58043 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58043&action=edit [SIGNED-OFF] Bug 17196: DB Changes -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58029|0 |1 is obsolete| | --- Comment #77 from Zeno Tajoli <z.tajoli@cineca.it> --- Created attachment 58045 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58045&action=edit [SIGNED-OFF] Bug 17196: Fix - Update metadata on update -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #79 from Zeno Tajoli <z.tajoli@cineca.it> --- Created attachment 58047 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58047&action=edit [SIGNED-OFF] Bug 17196: Fix build_oai_sets.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This develop moves marcxml release notes| |out of the biblioitems | |table and inserts it inside | |two new tables: | |biblio_metadata and | |deletedbiblio_metadata. SQL | |query on table biblioitems | |will boost on performance. | |Storing the marcxml in a | |specific table would allow | |use to store several kind | |of metadata (USMARC, | |MARCXML, MIJ, etc.) and | |different formats | |(marcflavour). | |ATTENTION: | |all reports that use XPATH | |to retrive MARC tags from | |biblioitems.marcxml need to | |be updated/rewritten -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |nick@bywatersolutions.com --- Comment #82 from Nick Clemens <nick@bywatersolutions.com> --- Testing this I found one things: When trying delete_records_via_leader.pl: kohadev-koha@kohadevbox:/home/vagrant/kohaclone$ perl misc/cronjobs/delete_records_via_leader.pl -v -c -i RECORD: 288 Can't use string ("Koha::Biblioitem") as a HASH ref while "strict refs" in use at /home/vagrant/kohaclone/Koha/Object.pm line 219. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #83 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 58504 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58504&action=edit Bug 17196: Fix delete_records_via_leader.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58040|0 |1 is obsolete| | --- Comment #85 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 58506 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58506&action=edit Bug 17196: Fix query builder for item search The item search needs to join on biblio_metadata to allow search on marcxml field Test plan: Launch complex item searches (using marc fields). Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58042|0 |1 is obsolete| | --- Comment #87 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 58508 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58508&action=edit Bug 17196: Update switch_marc21_series_info.pl Not sure if this script is still used, could someone confirm? Test plan: If you know how to test it, please do Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58044|0 |1 is obsolete| | --- Comment #89 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 58510 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58510&action=edit Bug 17196: Simplify tests for the different marc flavours centralize the way MARC field are set. Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58046|0 |1 is obsolete| | --- Comment #91 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 58512 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58512&action=edit Bug 17196: follow-up for bug 17088 If bug 17088 is pushed first, we will need this patch. Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58048|0 |1 is obsolete| | --- Comment #93 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 58514 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58514&action=edit Bug 17196: Update POD of sanitise_records.pl Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #95 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 58517 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58517&action=edit Bug 17196 - (QA Followup) Add items() to Koha::Biblioitems Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #97 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Still applies. If it does on early Friday morning, I will have a look. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58505|0 |1 is obsolete| | --- Comment #99 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58923 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58923&action=edit Bug 17196: Move marcxml out of the biblioitems table Two discussions on koha-devel lead to the same conclusion: biblioitems.marcxml should be moved out this table - biblio and biblioitems http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html - biblioitems.marcxml & biblioitems.marc / HUGE performance issue ! http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html There are several goals to do it: - Performance As Paul Poulain wrote, a simple query like SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear; takes more than 10min on a DB with more than 1M bibliographic records but only 3sec (!) on the same DB without the biblioitems.marcxml field Note that priori to this patch set, the biblioitems.marcxml was not retrieved systematically, but was, at least, in C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders - Flexibility Storing the marcxml in a specific table would allow use to store several kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats (marcflavour) - Clean code It would be a first step toward Koha::MetadataRecord for bibliographic records (not done in this patch set). Test plan: - Update the DBIC Schema - Add / Edit / Delete / Import / Export bibliographic records - Add items - Reindex records using ES - Confirm that the following scripts still work: * misc/cronjobs/delete_records_via_leader.pl * misc/migration_tools/build_oai_sets.pl - Look at the reading history at the OPAC (opac-readingrecord.pl) - At the OPAC, click on a tag, you must see the result Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108. Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58507|0 |1 is obsolete| | --- Comment #101 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58925 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58925&action=edit Bug 17196: Update misc/maintenance/sanitize_records.pl Test plan: Use the sanitize_records.pl maintenance script with the --auto-search option Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58509|0 |1 is obsolete| | --- Comment #103 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58927 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58927&action=edit Bug 17196: DB Changes Remove biblioitems.marcxml and deletedbiblioitems.marcxml and adds 2 table biblio_metadata and deletedbiblio_metadata Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58511|0 |1 is obsolete| | --- Comment #105 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58929 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58929&action=edit Bug 17196: Fix - Update metadata on update Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58513|0 |1 is obsolete| | --- Comment #107 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58931 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58931&action=edit Bug 17196: Fix build_oai_sets.pl The wrong value was retrieved. Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58516|0 |1 is obsolete| | --- Comment #109 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58933 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58933&action=edit Bug 17196: Fix delete_records_via_leader.pl Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #111 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58935 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58935&action=edit Bug 17196: [QA Follow-up] Correct POD in ILSDI/Services.pm QA tools complains about: Apparent command =cut not preceded by blank line. Trivial fix. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #113 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58937 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58937&action=edit Bug 17196: [QA Follow-up] Wrong module in Record.t Should be the singular version instead of the plural one. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #115 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Still working on it little bit, but looks good. Please wait.. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #116 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Some preliminary findings: opac/opac-readingrecord.pl: Calling GetXmlBiblio seems useless here. Probably should be GetMarcBiblio. opac-search.pl: map { (C4::Context->config('zebra_bib_index_mode') eq 'dom')? $_->{marcxml}: $_->{marc}; } This marc reference should not have been there anymore? C4/Items.pm: changes aimed for catalogue/itemsearch.pl, but I am not sure if they are needed. Does item search actually looks into the xml? Seems not. New report?? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #118 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #117)

...

C4/Items.pm: changes aimed for catalogue/itemsearch.pl, but I am not sure if they are needed. Does item search actually looks into the xml? Seems not. New report??

I don't think so, it uses it.

See C4::Items::_SearchItems_build_where_fragment -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #120 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- dmin -> admin -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58939|0 |1 is obsolete| | --- Comment #122 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58943 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58943&action=edit Bug 17196: Remove occurrence of marcxml in columns.def Also remove a wrong ref to the marc column Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #124 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 58947 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58947&action=edit Bug 17196: [QA Follow-up] Additional fix on acqui/basketgroup acqui/basketgroup calls GetOrders and expects marcxml in the results. Fixing it by an additional call of GetXmlBiblio. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #126 from Josef Moravec <josef.moravec@gmail.com> --- (In reply to Marcel de Rooy from comment #125)

Last comments:

...

Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108. Failed QA several weeks ago. The author did not respond until now.

Ere Maijala is primerly working on VuFind development, so probably we could try to contact them by e-mail directly to ask if he is plannig to work a bit on this OAI enhancement. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | Patch complexity|--- |Medium patch -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |17898 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17898 [Bug 17898] Add a way to automatically convert SQL reports -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #132 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I am willing to change the names of these fields if you have a good suggestion. It's the beginning of the end of MARC, so I think I can spend 10 min to rename the fields. (In reply to David Cook from comment #130)

I don't know. I'm guessing all the work's actually be done on this one anyway, so I'm probably too late with these comments.

Indeed, I asked for feedbacks on these patches for the last 6 months... But better late than never, isn't it? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |olli-antti.kivilahti@jns.fi --- Comment #134 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- I don't want to beat a dead horse, but can't resist:

There are several goals to do it: - Performance As Paul Poulain wrote, a simple query like SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear; takes more than 10min on a DB with more than 1M bibliographic records but only 3sec (!) on the same DB without the biblioitems.marcxml field

All our test environments: MariaDB [koha]> SELECT COUNT(*) FROM biblioitems; +----------+ | COUNT(*) | +----------+ | 1424537 | +----------+ 1 row in set (0.34 sec) On Ubuntu14.04 MariaDB using a normal 10k HDD with Intel Xeon 3GHz: innodb_buffer_pool_size = 128MB MariaDB [koha]> SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear; +-----------------+------------------------+ | publicationyear | count(publicationyear) | +-----------------+------------------------+ | NULL | 0 | +-----------------+------------------------+ 1 row in set (39.64 sec) The search is arguably slow. I wonder how the performance inspiration server (10 min search time) is configured? On Ubuntu14.04 MariaDB using RAID-1 SSD with Intel Xeon 3GHz 32GB RAM innodb_buffer_pool_size = 24G (Percona autoconfigurer) MariaDB [koha]> SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear; +-----------------+------------------------+ | publicationyear | count(publicationyear) | +-----------------+------------------------+ | NULL | 0 | +-----------------+------------------------+ 1 row in set (5.56 sec) RAM and SSDs are really cheap nowadays. See also Bug 18265 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #135 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Olli-Antti Kivilahti from comment #134)

I don't want to beat a dead horse, but can't resist:

Try to be around when we ask for feedbacks and tests, *before* things are pushed. I am sure it would be more useful for everybody. I cannot count how many times I have asked for help on this one.

The search is arguably slow. I wonder how the performance inspiration server (10 min search time) is configured?

"This is a test DB, not optimized, but we were surprised by the results." http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 --- Comment #137 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Followup Bug 18265 is ready to be SO'd. Regardless of the bugs found, I think this change is welcome. I hope we get the fixes in asap. http://quoteinvestigator.com/2014/12/16/no-mistakes/ -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|This development moves |This development moves release notes|marcxml out of the |marcxml out of the |biblioitems and |biblioitems and |deletedbiblioitems tables |deletedbiblioitems tables |and moves it to two new |and moves it to two new |tables: biblio_metadata and |tables: biblio_metadata and |deletedbiblio_metadata. SQL |deletedbiblio_metadata. SQL |queries on the marcxml data |queries using the |will get a performance |biblioitems table but not |boost. Storing the marcxml |the marcxml column will get |in a specific table will |a performance boost. |allow us to store several |Storing the marcxml in a |metadata formats (USMARC, |specific table will allow |MARCXML, MIJ, etc.). |us to store several |ATTENTION: all reports that |metadata formats (USMARC, |use ExtractValue to |MARCXML, MIJ, etc.). |retrieve MARC tags from |ATTENTION: all reports that |biblioitems.marcxml need to |use ExtractValue to |be updated/rewritten. |retrieve MARC tags from | |biblioitems.marcxml need to | |be updated/rewritten. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |19332 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19332 [Bug 19332] Basket grouping PDF and CSV exports empty -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |19583 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19583 [Bug 19583] Report updater triggers on auth_header.marcxml -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |21908 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21908 [Bug 21908] Add biblio_metadata to rebuild_zebra.pl tables -- You are receiving this mail because: You are watching all bug changes.