[Bug 23828] New: Elasticsearch - ES - Authority record results not ordered correctly
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Bug ID: 23828 Summary: Elasticsearch - ES - Authority record results not ordered correctly Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com For example: Catholic Church Addresses, essays, lectures Catholic Church El Salvador Addresses, essays, lectures . . . Catholic Church. The 'El Salvador' displays first, but is sorted as coming after Address... Catcholic hcurch with an ending period comes after Catholic Church with subfields We should ideally generate the sort fields in the same order that fields are listed in the record - that seems to be how the display of results is formed -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mkstephens@lancasterseminar | |y.edu --- Comment #1 from Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> --- Part of this issue is resolved by tweaking the mappings for authorities in ES. The current community authority mappings are indexing MARC fields and subfields as strings. Example: 110(abcdefghjklnpqstvxyz) is mapped to Heading. When "Catholic Church" is used for "Search main heading," the result list looks like this (subfields added for clarity): - $a Catholic Church $x Clergy $v Biography - $a Catholic Church $z Italy $x Clergy $v Biography - $a Catholic Church $x Doctrines - $a Catholic Church $x Fiction - $a Catholic Church $v Pastoral letters and charges - $a Catholic Church $x Adult Education $v Periodicals It is ordering them this way because they are indexed with the subfields in alphabetical order -- despite the fact that the subfields do not appear in alphabetical order in the record. When 110 is mapped to Heading, without the (abcd...), the result list looks like this: - Catholic Church Adult Education Periodicals - Catholic Church Adult Education United States - Catholic Church Clergy Biography - Catholic Church Doctrines - Catholic Church Fiction - Catholic Church Italy Clergy Biography WHAT IS STILL A BUG > If a heading has a period in it, it will still appear after all the headings without a period. In the Catholic Church example, some authority records use a period: Catholic Church. Archdiocese of San Salvador (El Salvador). Archbishop (1977-1980 : Romero) That particular heading and all others that follow a similar format will not appear in the results list alphabetically. It appears at the end after all "Catholic Church" headings that do not use a period. We should probably be escaping punctuation when searching and/or indexing authorities. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Heather <heather_hernandez@nps.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |heather_hernandez@nps.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 110175 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110175&action=edit Bug 23828: Unit tests -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 110176 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110176&action=edit Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Note: these patches do not address punctuation, we should file a new bug for that issue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- Note: these patches do not address punctuation, we should file a new bug for that issue -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #6 from Heather <heather_hernandez@nps.gov> --- I put in Bug 26472 to address punctuation in general in the filing/ordering of authorities search results, having just come across a good example of the problem in our catalog!:) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #110175|0 |1 is obsolete| | --- Comment #7 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 110547 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110547&action=edit Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #110176|0 |1 is obsolete| | --- Comment #8 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 110548 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110548&action=edit Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- There are lots of merge markers in the test file - this hasn't applied cleanly when tested. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #110547|0 |1 is obsolete| | Attachment #110548|0 |1 is obsolete| | --- Comment #10 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 111329 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111329&action=edit Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Bug 23828: (follow-up) fix unit test merge -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #11 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 111330 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111330&action=edit Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #12 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Katrin Fischer from comment #9)
There are lots of merge markers in the test file - this hasn't applied cleanly when tested.
Only the unit tests failed, the testing should still be valid as the code itself applied and worked -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111329|0 |1 is obsolete| | --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 111813 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111813&action=edit Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Bug 23828: (follow-up) fix unit test merge Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111330|0 |1 is obsolete| | --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 111814 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111814&action=edit Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 111815 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111815&action=edit Bug 23828: (QA follow-up) Fix number of tests Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111814|0 |1 is obsolete| | --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 112142 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112142&action=edit Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings Currently if you combine subfields in the marc mappings the subfields are indexed in the order listed in the mapping. i.e. 650(avxyz) in mapping and in record: 650 $aHeading $zGeosubdiv $vFormsubdiv is indexed as: Heading Formsubdiv Geosubdiv We should preserve the order and index as: Heading Geosubdiv Formsubdiv We can use built in function in Marc::Field to achieve this To test: 1 - It is easy to find examples of this using authorities 2 - Find or create a record with subfields order azv e.g. 150$aActresses$zUnited states$vBiography 3 - Add or have a second authority e.g. 150$aActresses$vPortraits 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz) find at: Administration->Search engine configuration (Elasticsearch)->Authorities tab 5 - Index the records in Elasticsearch perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692 6 - View the first record in the ES index curl es:9200/koha_kohadev_authorities/data/1692?pretty 7 - Note 'Heading' field is ordered as in the mapping 8 - Search authorities for 'contains' "act" 9 - Note the records sort incorrectly 10 - Apply patches 11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692 12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty 13 - Note the order is now preserved 14 - Search authorities for 'contains' "act" 15 - Note the records sort correctly Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> JD amended patch: Fix FAIL spelling combind ==> combined -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111813|0 |1 is obsolete| | Attachment #111815|0 |1 is obsolete| | --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 112143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112143&action=edit Bug 23828: Unit tests Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Bug 23828: (follow-up) fix unit test merge Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 23828: (QA follow-up) Fix number of tests Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 --- Comment #18 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|20.11.00 |20.11.00, 20.05.06 released in| | Status|Pushed to master |Pushed to stable CC| |lucas@bywatersolutions.com --- Comment #19 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 20.05.x for 20.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz --- Comment #20 from Aleisha Amohia <aleisha@catalyst.net.nz> --- enhancement, not backported to 19.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23828 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Pushed to stable |RESOLVED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org