[Bug 24807] New: Add "year" type to improve sorting by publication date
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Bug ID: 24807 Summary: Add "year" type to improve sorting by publication date Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: glasklas@gmail.com When sorting on date-of-publication (a year, field 008, positions 7-10), the current behavior when the field contains an invalid year (containing word characters like "uuuu" or just whitespace), these results appear first when sorting in descending order. Invalid values like these should really not be indexed at all. If so (having null values instead) they will always appear last when sorting, regardless of sort order. This fix adds a "year" search field type, that will reject any non-year values, resulting in a more user friendly sorting behavior. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #1 from David Gustafsson <glasklas@gmail.com> --- Created attachment 100170 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100170&action=edit Bug 24807: Add "year" type to improve sorting behaviour Add a "year" search field type. Fields with this type will only retain values that looks like years, so invalid values such as whitespace or word characters will not be indexed. This for instance improves the behaviour when sorting by "date-of-publication". If all values are indexed, records with junk data instead of valid years will appear first among the search results, drowing out relevant hits. If assigning this field the "year" type these records will instead always appear last, regarless of sort order. To test: 1) Have at least two biblios, one with a valid year in 008 (pos 7-10) and another with an invalid one ("uuuu" for example) 2) Perform a wildcard search (*) and sort results by publication date. 3) The record with invalid year of pulication in 008 should appear first 4) Apply patch and run database updates 5) Reindex ElasticSearch 6) Perform the same search as in 2) 7) The record with the invalid year should now appear last -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100170|0 |1 is obsolete| | --- Comment #2 from David Gustafsson <glasklas@gmail.com> --- Created attachment 100171 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100171&action=edit Bug 24807: Add "year" type to improve sorting behaviour Add a "year" search field type. Fields with this type will only retain values that looks like years, so invalid values such as whitespace or word characters will not be indexed. This for instance improves the behaviour when sorting by "date-of-publication". If all values are indexed, records with junk data instead of valid years will appear first among the search results, drowning out more relevant hits. If assigning this field the "year" type these records will instead always appear last, regarless of sort order. To test: 1) Have at least two biblios, one with a valid year in 008 (pos 7-10) and another with an invalid one ("uuuu" for example) 2) Perform a wildcard search (*) and sort results by publication date. 3) The record with invalid year of pulication in 008 should appear first 4) Apply patch and run database updates 5) Reindex ElasticSearch 6) Perform the same search as in 2) 7) The record with the invalid year should now appear last -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Assignee|koha-bugs@lists.koha-commun |glasklas@gmail.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #3 from David Gustafsson <glasklas@gmail.com> --- Created attachment 100176 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100176&action=edit Bug 24807: Add database update script -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |black23@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24559 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 102547 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102547&action=edit Bug 24807: Update tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 102548 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102548&action=edit Bug 24807: Add suppport for uncertain fields and ranges To test: 1 - Have some records with uncertain dates in the 008 19uu, 195u, etc. 2 - Index them in Elasticsearch 3 - Do a search that will return them 4 - Sort results by publication/copyright date 5 - Note odd results 6 - Apply patch 7 - Reindex 8 - Sorting should be improved -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #6 from Nick Clemens <nick@bywatersolutions.com> --- Hi David, I like your patch better than mine (bug 24559) but I do think we should support uncertain dates and ranges in these fields as they do meet MARC standards. 195u isn't exactly 1950, but if looking for a range from 1940-1960 I would expect it to return so I think indexing as 1950 is reasonable In testing your code did not work with dates like c1964 I added tests and made some adjustment. The array stuff may need work, depending on if we would support being a year field and having non-filing characters, but I can't see a reason to do that. Please let me know what you think of these patches. -Nick -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #7 from David Gustafsson <glasklas@gmail.com> --- Hi! Sorry about the late reply. Thank you very much for considering my patch. I am a little bit unfamiliar with conventions with regards to marc record dates and did not anything like c1964 in my testing data, is "c" in this case for copyright? Is "u" some kind of convention that should have a special case? I think there is another callback option "value_callbacks" that can be used for preprocessing values before filtering, for example replace "u" with "0", and strip other characters. I can update the patch using this approach later today, I think it is a little bit semantically to perform filtering and transformation in different steps. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #8 from David Gustafsson <glasklas@gmail.com> --- I get: error: sha1 information is lacking or useless (t/Koha/SearchEngine/Elasticsearch.t). error: could not build fake ancestor Patch failed at 0001 Bug 24807: Update tests When applying the patch. I think the first two commits needs to be rebased since they do not apply anymore. The last two seems to be using a different base. I might be able to work this out, but will have to wait for tomorrow. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #9 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to David Gustafsson from comment #7)
Hi! Sorry about the late reply. Thank you very much for considering my patch. I am a little bit unfamiliar with conventions with regards to marc record dates and did not anything like c1964 in my testing data, is "c" in this case for copyright? Is "u" some kind of convention that should have a special case? I think there is another callback option "value_callbacks" that can be used for preprocessing values before filtering, for example replace "u" with "0", and strip other characters. I can update the patch using this approach later today, I think it is a little bit semantically to perform filtering and transformation in different steps.
There are more fields than just the 008 that can contain 'year' - pubdate and copydate are the main two used in koha 008/264c/260c The u is unknown, it is documented in the MARC: https://www.loc.gov/marc/bibliographic/bd008a.html search for (195u) The c is copyright we can also have p or ranges etc., documented here: https://www.oclc.org/bibformats/en/2xx/264.html#subfieldc -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100171|0 |1 is obsolete| | --- Comment #10 from David Gustafsson <glasklas@gmail.com> --- Created attachment 103084 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103084&action=edit Bug 24807: Add "year" type to improve sorting behaviour Add a "year" search field type. Fields with this type will only retain values that looks like years, so invalid values such as whitespace or word characters will not be indexed. This for instance improves the behaviour when sorting by "date-of-publication". If all values are indexed, records with junk data instead of valid years will appear first among the search results, drowning out more relevant hits. If assigning this field the "year" type these records will instead always appear last, regarless of sort order. To test: 1) Have at least two biblios, one with a valid year in 008 (pos 7-10) and another with an invalid one ("uuuu" for example) 2) Perform a wildcard search (*) and sort results by publication date. 3) The record with invalid year of pulication in 008 should appear first 4) Apply patch and run database updates 5) Reindex ElasticSearch 6) Perform the same search as in 2) 7) The record with the invalid year should now appear last -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100176|0 |1 is obsolete| | --- Comment #11 from David Gustafsson <glasklas@gmail.com> --- Created attachment 103085 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103085&action=edit Bug 24807: Add database update script -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #102547|0 |1 is obsolete| | --- Comment #12 from David Gustafsson <glasklas@gmail.com> --- Created attachment 103086 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103086&action=edit Bug 24807: Update tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #102548|0 |1 is obsolete| | --- Comment #13 from David Gustafsson <glasklas@gmail.com> --- Created attachment 103087 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103087&action=edit Bug 24807: Add suppport for uncertain fields and ranges To test: 1 - Have some records with uncertain dates in the 008 19uu, 195u, etc. 2 - Index them in Elasticsearch 3 - Do a search that will return them 4 - Sort results by publication/copyright date 5 - Note odd results 6 - Apply patch 7 - Reindex 8 - Sorting should be improved -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #14 from David Gustafsson <glasklas@gmail.com> --- Ok! Thanks for the references, will have a look at it. Rebased against latest master so that applies again. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #15 from David Gustafsson <glasklas@gmail.com> --- At first did not realize that one case is split up one subfield value into multiple ones. The way filter_callbacks works (it might be a bit overkill to allow multiple callbacks, but that is the way it is right now), is that the value returned is passed on to the next callback. So if you return an array the next callback (if one would exist) would expect a scalar and crash. I found a way around this by creating a new kind of callback, "tokenize_callbacks", so that the rest of the callback can keep their current function signature. Will be back with patch to demonstrate this in a while. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #16 from David Gustafsson <glasklas@gmail.com> --- On another note all sort fields are currently concatenated in the end, so "1963-2003" will end up as "1963 2003" in Elastic anyway. I'm not sure that I agree with this since sorting on numeric fields with multiple values actually can make sense (if sorting in descending order the highest value will be used by elastic and the other way around). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #17 from David Gustafsson <glasklas@gmail.com> --- We do this in the Perl code to be more specific, so could easily be "fixed". -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #18 from David Gustafsson <glasklas@gmail.com> --- Created attachment 103163 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103163&action=edit Bug 24807: Refactor using tokenize_callbacks -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #19 from David Gustafsson <glasklas@gmail.com> --- Created attachment 103277 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103277&action=edit Bug 24807: Simplify with new and imporved value_callbacks -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #20 from David Gustafsson <glasklas@gmail.com> --- Realized that all of what tokenize_callbacks and filter_callbacks do could be expressed as value_callback if changing the processing of that callback to the one used by tokenize_callbacks (if effect allowing value callbacks to return a scalar, a list (replacing tokenize_callbacks) or an empty list (replacing filter_callbacks). So cleaned up the code with regards to this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #21 from Nick Clemens <nick@bywatersolutions.com> --- *** Bug 24559 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103084|0 |1 is obsolete| | Attachment #103085|0 |1 is obsolete| | Attachment #103086|0 |1 is obsolete| | Attachment #103087|0 |1 is obsolete| | Attachment #103163|0 |1 is obsolete| | Attachment #103277|0 |1 is obsolete| | --- Comment #22 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 104525 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104525&action=edit Bug 24807: Add "year" type to improve sorting behaviour Add a "year" search field type. Fields with this type will only retain values that looks like years, so invalid values such as whitespace or word characters will not be indexed. This for instance improves the behaviour when sorting by "date-of-publication". If all values are indexed, records with junk data instead of valid years will appear first among the search results, drowning out more relevant hits. If assigning this field the "year" type these records will instead always appear last, regarless of sort order. To test: 1) Have at least two biblios, one with a valid year in 008 (pos 7-10) and another with an invalid one ("uuuu" for example) 2) Perform a wildcard search (*) and sort results by publication date. 3) The record with invalid year of pulication in 008 should appear first 4) Apply patch and run database updates 5) Reindex ElasticSearch 6) Perform the same search as in 2) 7) The record with the invalid year should now appear last Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #23 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 104526 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104526&action=edit Bug 24807: Add database update script Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #24 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 104527 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104527&action=edit Bug 24807: Update tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #25 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 104528 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104528&action=edit Bug 24807: Add suppport for uncertain fields and ranges To test: 1 - Have some records with uncertain dates in the 008 19uu, 195u, etc. 2 - Index them in Elasticsearch 3 - Do a search that will return them 4 - Sort results by publication/copyright date 5 - Note odd results 6 - Apply patch 7 - Reindex 8 - Sorting should be improved Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #26 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 104529 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104529&action=edit Bug 24807: Refactor using tokenize_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #27 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 104530 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104530&action=edit Bug 24807: Simplify with new and imporved value_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #28 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 104531 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104531&action=edit Bug 24807: (follow-up) Fix spelling Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |normal -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #29 from Julian Maurice <julian.maurice@biblibre.com> ---
return map { s/[u]/0/gr } ( $value =~ /[0-9u]{4}/g );
This line has some problems IMO: - It only considers years of 4 digits. I believe it is valid in MARC21 to write years with 3 digits or less, so it should be possible to index those values too. - In UNIMARC uncertain digits can be replaced by a blank (space) in some cases (100$a for instance : https://www.transition-bibliographique.fr/wp-content/uploads/2018/07/B100-6-... [in french, sorry]) - Replacing the uncertain character by 0 can be problematic. 197u is not 1970. Ideally, searching for records published in 1975 should return records where publication date is "197u", right ? Elasticsearch has an integer_range data type that could be useful in this situation. What do you think ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #30 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Julian Maurice from comment #29)
return map { s/[u]/0/gr } ( $value =~ /[0-9u]{4}/g );
This line has some problems IMO: - It only considers years of 4 digits. I believe it is valid in MARC21 to write years with 3 digits or less, so it should be possible to index those values too. - In UNIMARC uncertain digits can be replaced by a blank (space) in some cases (100$a for instance : https://www.transition-bibliographique.fr/wp-content/uploads/2018/07/B100-6- 2010.pdf [in french, sorry])
From what little I understand there, UNIMARC uses '#' as the unkown character? or '#' translates to blank?
leading blanks should be preserved as 0? and trailing ranged as you say below?
- Replacing the uncertain character by 0 can be problematic. 197u is not 1970. Ideally, searching for records published in 1975 should return records where publication date is "197u", right ? Elasticsearch has an integer_range data type that could be useful in this situation. What do you think ?
That is interesting, how does the range affect sorting? Might it still be worthwhile to sort unknown/ranged dates either at start or end, so file the sort value as either 1970 or 1979? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 anne-cecile.bories@iepg.fr changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |anne-cecile.bories@iepg.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #31 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Nick Clemens from comment #30)
From what little I understand there, UNIMARC uses '#' as the unkown character? or '#' translates to blank? I think that '#' translates to blank. UNIMARC cataloguing plugin for 100$a in Koha uses spaces.
leading blanks should be preserved as 0? I think so.
and trailing ranged as you say below? I don't know...
- Replacing the uncertain character by 0 can be problematic. 197u is not 1970. Ideally, searching for records published in 1975 should return records where publication date is "197u", right ? Elasticsearch has an integer_range data type that could be useful in this situation. What do you think ?
That is interesting, how does the range affect sorting? Might it still be worthwhile to sort unknown/ranged dates either at start or end, so file the sort value as either 1970 or 1979?
I have not tested integer_range yet. Rethinking about this, maybe replacing "uncertain" by 0 is good enough for now. Support for uncertain dates as ranges can be added later. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #32 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 106768 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=106768&action=edit Bug 24807: (follow-up) Add support for spaces as unknown characters -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply --- Comment #33 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Needs a small rebase. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104525|0 |1 is obsolete| | --- Comment #34 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108382 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108382&action=edit Bug 24807: Add "year" type to improve sorting behaviour Add a "year" search field type. Fields with this type will only retain values that looks like years, so invalid values such as whitespace or word characters will not be indexed. This for instance improves the behaviour when sorting by "date-of-publication". If all values are indexed, records with junk data instead of valid years will appear first among the search results, drowning out more relevant hits. If assigning this field the "year" type these records will instead always appear last, regarless of sort order. To test: 1) Have at least two biblios, one with a valid year in 008 (pos 7-10) and another with an invalid one ("uuuu" for example) 2) Perform a wildcard search (*) and sort results by publication date. 3) The record with invalid year of pulication in 008 should appear first 4) Apply patch and run database updates 5) Reindex ElasticSearch 6) Perform the same search as in 2) 7) The record with the invalid year should now appear last Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104526|0 |1 is obsolete| | --- Comment #35 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108383 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108383&action=edit Bug 24807: Add database update script Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104527|0 |1 is obsolete| | --- Comment #36 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108384 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108384&action=edit Bug 24807: Update tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104528|0 |1 is obsolete| | --- Comment #37 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108385 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108385&action=edit Bug 24807: Add suppport for uncertain fields and ranges To test: 1 - Have some records with uncertain dates in the 008 19uu, 195u, etc. 2 - Index them in Elasticsearch 3 - Do a search that will return them 4 - Sort results by publication/copyright date 5 - Note odd results 6 - Apply patch 7 - Reindex 8 - Sorting should be improved Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104529|0 |1 is obsolete| | --- Comment #38 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108386 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108386&action=edit Bug 24807: Refactor using tokenize_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104530|0 |1 is obsolete| | --- Comment #39 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108387 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108387&action=edit Bug 24807: Simplify with new and imporved value_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104531|0 |1 is obsolete| | --- Comment #40 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108388 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108388&action=edit Bug 24807: (follow-up) Fix spelling Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #106768|0 |1 is obsolete| | --- Comment #41 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108389 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108389&action=edit Bug 24807: (follow-up) Add support for spaces as unknown characters -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #42 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am still learning about Elasticsearch. In order to test this patch set, I applied the patches and tried do reindex. When I run the reindex with -d, I have an error that doesn't appear without the patches: kohadev-koha@kohadevbox:/home/vagrant/kohaclone$ ./misc/search_tools/rebuild_elasticsearch.pl -d One or more ElasticSearch errors occurred when indexing documents at ./misc/search_tools/rebuild_elasticsearch.pl line 302. The sorting appears to work really well. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #43 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Katrin Fischer from comment #42)
I am still learning about Elasticsearch. In order to test this patch set, I applied the patches and tried do reindex.
When I run the reindex with -d, I have an error that doesn't appear without the patches:
kohadev-koha@kohadevbox:/home/vagrant/kohaclone$ ./misc/search_tools/rebuild_elasticsearch.pl -d One or more ElasticSearch errors occurred when indexing documents at ./misc/search_tools/rebuild_elasticsearch.pl line 302.
The sorting appears to work really well.
Record #115 "Perl hacks" has a malformed 008 - this causes the error. The error does not occur when using ES6. I think this is okay as we expect errors if records have issues. We can use compare_es_to_db.pl to identify the records, or see bug 26310 for option to get more debug info -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108382|0 |1 is obsolete| | --- Comment #44 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 109997 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109997&action=edit Bug 24807: Add "year" type to improve sorting behaviour Add a "year" search field type. Fields with this type will only retain values that looks like years, so invalid values such as whitespace or word characters will not be indexed. This for instance improves the behaviour when sorting by "date-of-publication". If all values are indexed, records with junk data instead of valid years will appear first among the search results, drowning out more relevant hits. If assigning this field the "year" type these records will instead always appear last, regarless of sort order. To test: 1) Have at least two biblios, one with a valid year in 008 (pos 7-10) and another with an invalid one ("uuuu" for example) 2) Perform a wildcard search (*) and sort results by publication date. 3) The record with invalid year of pulication in 008 should appear first 4) Apply patch and run database updates 5) Reindex ElasticSearch 6) Perform the same search as in 2) 7) The record with the invalid year should now appear last Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108383|0 |1 is obsolete| | --- Comment #45 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 109998 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109998&action=edit Bug 24807: Add database update script Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108384|0 |1 is obsolete| | --- Comment #46 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 109999 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109999&action=edit Bug 24807: Update tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108385|0 |1 is obsolete| | --- Comment #47 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 110000 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110000&action=edit Bug 24807: Add suppport for uncertain fields and ranges To test: 1 - Have some records with uncertain dates in the 008 19uu, 195u, etc. 2 - Index them in Elasticsearch 3 - Do a search that will return them 4 - Sort results by publication/copyright date 5 - Note odd results 6 - Apply patch 7 - Reindex 8 - Sorting should be improved Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108386|0 |1 is obsolete| | --- Comment #48 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 110001 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110001&action=edit Bug 24807: Refactor using tokenize_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108387|0 |1 is obsolete| | --- Comment #49 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 110002 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110002&action=edit Bug 24807: Simplify with new and imporved value_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108388|0 |1 is obsolete| | --- Comment #50 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 110003 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110003&action=edit Bug 24807: (follow-up) Fix spelling Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108389|0 |1 is obsolete| | --- Comment #51 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 110004 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110004&action=edit Bug 24807: (follow-up) Add support for spaces as unknown characters Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |20.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #52 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #53 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- There is a failing test, please fix ASAP t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t .. 1/3 # Looks like you planned 2 tests but ran 1. # Failed test 'update_index() tests' # at t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t line 98. Not a HASH reference at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 120. # Looks like your test exited with 255 just after 3. https://jenkins.koha-community.org/job/Koha_Master_U20/92/consoleFull -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #54 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 110355 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110355&action=edit Bug 24807: (QA follow-up) Remove uneccessary tests These tests fail now, the code expects a real response from ES in Indexer.pm but these tests mock 'bulk' and so don't have the necessary fields. We are testing the same code above and can just add the _id == biblionumber test -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=20492 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #55 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Follow-up pushed to master, thanks Nick! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 --- Comment #56 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 110884 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110884&action=edit Bug 24807: [20.05.x] Add "year" type to improve sorting behaviour Add a "year" search field type. Fields with this type will only retain values that looks like years, so invalid values such as whitespace or word characters will not be indexed. This for instance improves the behaviour when sorting by "date-of-publication". If all values are indexed, records with junk data instead of valid years will appear first among the search results, drowning out more relevant hits. If assigning this field the "year" type these records will instead always appear last, regarless of sort order. To test: 1) Have at least two biblios, one with a valid year in 008 (pos 7-10) and another with an invalid one ("uuuu" for example) 2) Perform a wildcard search (*) and sort results by publication date. 3) The record with invalid year of pulication in 008 should appear first 4) Apply patch and run database updates 5) Reindex ElasticSearch 6) Perform the same search as in 2) 7) The record with the invalid year should now appear last Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: Add database update script Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: Update tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: Add suppport for uncertain fields and ranges To test: 1 - Have some records with uncertain dates in the 008 19uu, 195u, etc. 2 - Index them in Elasticsearch 3 - Do a search that will return them 4 - Sort results by publication/copyright date 5 - Note odd results 6 - Apply patch 7 - Reindex 8 - Sorting should be improved Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: Refactor using tokenize_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: Simplify with new and imporved value_callbacks Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: (follow-up) Fix spelling Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: (follow-up) Add support for spaces as unknown characters Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Bug 24807: (QA follow-up) Remove uneccessary tests These tests fail now, the code expects a real response from ES in Indexer.pm but these tests mock 'bulk' and so don't have the necessary fields. We are testing the same code above and can just add the _id == biblionumber test -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Pushed to master |Pushed to stable Version(s)|20.11.00 |20.11.00, 20.05.05 released in| | --- Comment #57 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 20.05.x for 20.05.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz Severity|normal |enhancement --- Comment #58 from Aleisha Amohia <aleisha@catalyst.net.nz> --- enhancement, not backported to 19.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |36902 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 [Bug 36902] Blank year fields should not be indexed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43436 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43436 [Bug 43436] Elasticsearch doesn't order publication dates with 'u' correctly -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org