[Bug 36902] New: Blank year fields should not be indexed.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 Bug ID: 36902 Summary: Blank year fields should not be indexed. Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: andreas.jonsson@kreablo.se The comment says 'Only accept years containing digits and "u"' but proceeds to also accept fields with blanks: # Only accept years containing digits and "u" push @{$default_options->{value_callbacks}}, sub { my ($value) = @_; # Replace "u" with "0" for sorting return map { s/[u\s]/0/gr } ( $value =~ /[0-9u\s]{4}/g ); }; If the field contains blanks it is either because it is invalid or because the year is missing, so the field should not be indexed. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 --- Comment #1 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- Created attachment 166951 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=166951&action=edit Bug 36902: Filter year fields containing blanks Test plan: * If you are using koha-testing-docker, make sure that you have an elasticsearch container running (e.g., ku-es7) and that the searchengine system preference is set to ElasticSearch, and that the indexer is running (koha-es-indexer --start kohadev). * Under Koha administration -> Search engine configuration, make sure that the type of the field date-of-publication is set to 'year'. * If you needed to changed the type above, rebuild the index: koha-elasticsearch --delete --rebuild kohadev * Edit any record and change control field 008 position 7-10 (Date 1) to all blanks. * Save the record. * Show the elasticsearch record (Klick on "Show" in Elasticsearch record: Show") * Make sure there are no value "0000" for date-of-publication. * Edit the record again and change control field 008 position 7-10 (Date 1) to valid year. * Save the record. * Show the elasticsearch record. * Make sure there set year is present in the field date-of-publication. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 Andreas Jonsson <andreas.jonsson@kreablo.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |trivial Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |januszop@gmail.com --- Comment #2 from Janusz Kaczmarek <januszop@gmail.com> --- I would agree that a blank in 008/7-10 is usually an invalid value, but not for 008/6 = 'b' (a very rare case). The current behavior is more forgiving which may a be good or may be bad think. My doubt is: how this patch would impact the sorting feature, i.e. when sorting according to the date of publication, where these invalid records will be placed not having 008/7-10 indexed? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 --- Comment #3 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- The default is to sort documents with missing sort field last, which seems reasonable to me. My client wants to have this fixed for this very reason, i.e., to not have the search results cluttered by records with invalid year in 008 when sorting on date of publication. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #166951|0 |1 is obsolete| | --- Comment #4 from David Nind <david@davidnind.com> --- Created attachment 167065 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167065&action=edit Bug 36902: Filter year fields containing blanks Test plan: * If you are using koha-testing-docker, make sure that you have an elasticsearch container running (e.g., ku-es7) and that the searchengine system preference is set to ElasticSearch, and that the indexer is running (koha-es-indexer --start kohadev). * Under Koha administration -> Search engine configuration, make sure that the type of the field date-of-publication is set to 'year'. * If you needed to changed the type above, rebuild the index: koha-elasticsearch --delete --rebuild kohadev * Edit any record and change control field 008 position 7-10 (Date 1) to all blanks. * Save the record. * Show the elasticsearch record (Klick on "Show" in Elasticsearch record: Show") * Make sure there are no value "0000" for date-of-publication. * Edit the record again and change control field 008 position 7-10 (Date 1) to valid year. * Save the record. * Show the elasticsearch record. * Make sure there set year is present in the field date-of-publication. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Assignee|koha-bugs@lists.koha-commun |andreas.jonsson@kreablo.se |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA Depends on| |24807 CC| |nick@bywatersolutions.com --- Comment #5 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- See bug 24807 comment 29: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807#c29 3 digit years and UNIMARC need to be considered here. YASP? Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24807 [Bug 24807] Add "year" type to improve sorting by publication date -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@libriotech.no --- Comment #6 from Magnus Enger <magnus@libriotech.no> --- Sorry if this is not relevant here. We are seeing searches ordered by publication date, where the first few records are sorted as expected, and then the rest seem to be in random order. When we check the "Elasticsearch record" for records that fall in the "random" category, the data for date-of-publication often looks something like this: "date-of-publication": [ " ", "2019" ], My hunch is that the first, blank string is from 008, and the second string is from 260c. And that only the first string is indexed, so records with a blank year in 008 are returned at the end of the list, in seemingly random order. Would it be possible to skip the first string here, if it is only four blank spaces, and consider the second string for indexing, if there is a second string? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 --- Comment #7 from Janusz Kaczmarek <januszop@gmail.com> --- I sounds strange--by default only 008/7-10 should be indexed with date-of-publication (for MARC 21). (In 260c there are other things like 'cop.' etc. that would make sorting and searching by date unpredictable and therefor 260c shouldn't be indexed with date-of-publication, IMO). I would check the local mapping. Or maybe there are two 008 fields in the record in question...? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36902 --- Comment #8 from Magnus Enger <magnus@libriotech.no> --- Ah, indexing 260c as date-of-publication was a local thing. Sorry for the noise. (The solution for us is probably to run a script to copy dates from 260c to 008/7-10) -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org