https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 62171 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=62171&action=edit Bug 18434 - Elasticsearch indexing broken with newer catmandu version To test: 1 - Make sure you have latest koha deps, catmandu versions should be: libcatmandu-marc-perl 1.09-1~kohadev1 libcatmandu-perl 1.0304-2~kohadev1 2 - Reindex elastic 3 - Try searching and likely notice odd results 4 - Try: curl -XGET 'http://localhost:9200/koha_kohadev_biblios/data/792?pretty=true' with a known biblionumber and notice some null fields 5 - Apply patch 6 - Reindex 7 - Note fields are populated and search works as expected -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- How could we test this change? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #62702|0 |1 is obsolete| | --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 63452 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63452&action=edit Bug 18434: Elasticsearch indexing broken with newer catmandu version To test: 1 - Make sure you have latest koha deps, catmandu versions should be: libcatmandu-marc-perl 1.09-1~kohadev1 libcatmandu-perl 1.0304-2~kohadev1 2 - Reindex elastic 3 - Try searching and likely notice odd results 4 - Try: curl -XGET 'http://localhost:9200/koha_kohadev_biblios/data/792?pretty=true' with a known biblionumber and notice some null fields 5 - Apply patch 6 - Reindex 7 - Note fields are populated and search works as expected Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #7 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Tomás Cohen Arazi from comment #6)

Created attachment 63453 [details] [review] Bug 18434: (followup) Remove unneeded comma in fixer rules

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

I forgot to metion: To test: - Run: $ prove t/db_dependent/Koha_Elasticsearch.t => SUCCESS: All should be green! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #9 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Chris Cormack from comment #8)

Should this be needs signoff, or signed off. Rather than Assigned?

It should be pqa imho, but nick was going to double check the tests. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #63451|0 |1 is obsolete| | --- Comment #10 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 63473 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63473&action=edit Bug 18434: Add tests for K:SE:E::get_fixer_rules This patch tries to introduce exhaustive tests for this class method. I didn't try to provide a regression test for the current bug per-se, but cover the current method behaviour as much as I could. (kidclamp) I added a quick test of _convert_marc_to_json to use the mocking here and illuminate what the change does, before the patches this should fail (fields are indexed in place of one another), after it should succeed (new indexed fields are appended). A minor bug is highlighted by this new tests, I'll provide a followup for it. To test: - Run: $ sudo koha-shell kohadev k$ de kohaclone k$ prove t/db_dependent/Koha_Elasticsearch.t => FAIL: The returned fixer rules are not the expected ones Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #63452|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #11 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 63477 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63477&action=edit Bug 18434: Add tests for K:SE:E::get_fixer_rules This patch tries to introduce exhaustive tests for this class method. I didn't try to provide a regression test for the current bug per-se, but cover the current method behaviour as much as I could. (kidclamp) I added a quick test of _convert_marc_to_json to use the mocking here and illuminate what the change does, before the patches this should fail (fields are indexed in place of one another), after it should succeed (new indexed fields are appended). A minor bug is highlighted by this new tests, I'll provide a followup for it. To test: - Run: $ sudo koha-shell kohadev k$ de kohaclone k$ prove t/db_dependent/Koha_Elasticsearch.t => FAIL: The returned fixer rules are not the expected ones Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |18610 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18610 [Bug 18610] ElasticSearch indexing of facets needs to be updated for newer Catmandu versions -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Package: libcatmandu-marc-perl Version: 1.09-1~kohadev1 Package: libcatmandu-perl Version: 1.0304-2~kohadev1 { author [ [0] [ [0] [ [0] "Author" ] ], [1] [ [0] [ [0] "Corp Author" ] ] ], author__facet [ [0] [ [0] "Author" ], [1] [ [0] "Corp Author" ] ], author__sort [ [0] [ [0] "Author" ], [1] [ [0] "Corp Author" ] ], author__suggestion { input [ [0] "Author", [1] "Corp Author" ] }, es_id undef, record [ [0] [ [0] "LDR", [1] undef, [2] undef, [3] "_", [4] " " ], [1] [ [0] 001, [1] undef, [2] undef, [3] "_", [4] 1234567 ], [2] [ [0] 020, [1] " ", [2] " ", [3] "a", [4] 1234567890123 ], [3] [ [0] 100, [1] " ", [2] " ", [3] "a", [4] "Author" ], [4] [ [0] 110, [1] " ", [2] " ", [3] "a", [4] "Corp Author" ], [5] [ [0] 245, [1] " ", [2] " ", [3] "a", [4] "Title" ] ] } -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #18 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- { author [ [0] [ [0] [ [0] "Author" ] ], [1] [ [0] [ [0] "Corp Author" ] ] ], author__facet [ [0] [ [0] [ [0] "Author" ] ], [1] [ [0] [ [0] "Corp Author" ] ] ], author__sort [ [0] [ [0] [ [0] "Author" ] ], [1] [ [0] [ [0] "Corp Author" ] ] ], author__suggestion { input [ [0] "Author", [1] "Corp Author" ] }, es_id undef, record [ [21/108] [0] [ [0] "LDR", [1] undef, [2] undef, [3] "_", [4] " " ], [1] [ [0] 001, [1] undef, [2] undef, [3] "_", [4] 1234567 ], [2] [ [0] 020, [1] " ", [2] " ", [3] "a", [4] 1234567890123 ], [3] [ [0] 100, [1] " ", [2] " ", [3] "a", [4] "Author" ], [4] [ [0] 110, [1] " ", [2] " ", [3] "a", [4] "Corp Author" ], [5] [ [0] 245, [1] " ", [2] " ", [3] "a", [4] "Title" ] ] } -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #20 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Nick Clemens from comment #19)

I am looking here, it seems we have some version issues - I sent an email to Mirko and going to talk about versions/packagin to see what we can do

I tested on a new kohadevbox - pulling Mirko's packaged versions everything seems to work for me now: sudo apt-get install koha-elasticsearch ||/ Name Version Architecture Description +++-=============================================-===========================-===========================-=============================================================================================== ii libcatmandu-marc-perl 1.09-1~kohadev1 all Catmandu modules for working with MARC data ii libcatmandu-perl 1.0304-2~kohadev1 all metadata toolkit ii libcatmandu-store-elasticsearch-perl 0.0507-1~kohadev1 all searchable store backed by Elasticsearch It must be another module somewhere causing the extra level in your conversion - can you test with a new clean devbox and see if the error persists? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 --- Comment #21 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 64773 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=64773&action=edit Bug 18434 - Followup fix tests for sorting and factes -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #63478|0 |1 is obsolete| | --- Comment #23 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 64787 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=64787&action=edit Bug 18434: Elasticsearch indexing broken with newer catmandu version To test: 1 - Make sure you have latest koha deps, catmandu versions should be: libcatmandu-marc-perl 1.09-1~kohadev1 libcatmandu-perl 1.0304-2~kohadev1 2 - Reindex elastic 3 - Try searching and likely notice odd results 4 - Try: curl -XGET 'http://localhost:9200/koha_kohadev_biblios/data/792?pretty=true' with a known biblionumber and notice some null fields 5 - Apply patch 6 - Reindex 7 - Note fields are populated and search works as expected Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #64386|0 |1 is obsolete| | --- Comment #25 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 64789 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=64789&action=edit Bug 18434 - Followup - same changes for sort and facet fields To test: 1 - Index some stuff with multiple fields defined for sorting i.e. Authorites - make heading sortable - default is 110a and 111a for heading - a record with 111a empty will make the sort field empty 2 - view the record: curl http://localhost:9200/koha_kohadev_authorities/data/30?pretty=true 3 - Note the blank field 4 - Apply patch 5 - Reindex 6 - Fields should be correctly populated Unit tests to follow (once I have the originals working for all) Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #27 from Tomás Cohen Arazi <tomascohen@gmail.com> --- The last two patches for this one overlap with my patches for bug 18610, this ones are ok and i'm closing the other as duplicate. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #29 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 17.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18434 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |katrin.fischer@bsz-bw.de Status|Pushed to Stable |RESOLVED --- Comment #31 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Doesn't apply cleanly, possibly because of other Elasticsearch related patches missing in 16.11.x. Please get in touch with me if you want to have this included. -- You are receiving this mail because: You are watching all bug changes.