[Bug 15541] New: Prevent normalization during matching/import process
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 Bug ID: 15541 Summary: Prevent normalization during matching/import process Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: MARC Bibliographic record staging/import Assignee: gmcharlt@gmail.com Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org By default, when you're using a matching rule during an import, the data from the incoming record for a match point is normalized using the following regex: $value =~ s/[.;:,\]\[\)\(\/'"]//g; $value =~ s/^\s+//; $value =~ s/\s+$//; $value =~ s/\s+/ /g; The first line removes all sorts of punctuation and other marks. The second removes leading spaces. The third removes trailing spaces. The fourth converts multiple spaces into a single space. -- While this might work well in many cases, it's a problem when you're trying to match using a URL. During matching, using CHR Zebra indexing, the following occurs: "http://libris.kb.se/resource/bib/219553" becomes "HTTPLIBRISKBSERESOURCEBIB219553" If you're using a URL index, it's stored in Zebra as "http://libris.kb.se/resource/bib/219553", so no match is made. If you're using a Phrase index, it's stored in Zebra as "http libris kb se resource bib 219553". If you're using a Word index, it's tokenized so that each little alphanumeric piece between punctuation parts is indexed separately. -- The solution seems to be to either prevent normalization all together, or to "replace punctuation with spaces" in accordance with the default word-phrase-utf.chr file. I'd have to review what would need to be done for ICU indexing... -- An additional problem is C4::Search::SimpleSearch, as it converts all colons (ie ":") into equal signs (ie "="). This is obviously a problem when you're searching for a URL as the URL is stored as "http://libris.kb.se/resource/bib/219553" but you're searching for "http=//libris.kb.se/resource/bib/219553". No match will be made. In this case, I think we could pass an additional flag to C4::Search::SimpleSearch asking it not to normalize the query. Since it's a simple search, we are often able to make the CCL qualifier choices before passing in the query argument, so this seems reasonable. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Prevent normalization |Create additional |during matching/import |normalizers for Record |process |Matching Rules -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- Here's my latest findings: Input: "http://libris.kb.se/resource/bib/219553" C4::Matcher::_normalize() = "HTTPLIBRISKBSERESOURCEBIB219553" Zebra CHR = "http libris kb se resource bib 219553" Zebra ICU = "http libriskbse resource bib 219553" It seems to me that the smartest thing to do is NOT to normalize with C4::Matcher::_normalize(), because we're probably going to get it wrong as we have above. Zebra indexes "http://libris.kb.se/resource/bib/219553" as "http libris kb se resource bib 219553" (CHR Phrase) or as "http libriskbse resource bib 219553" (ICU Phrase) or as "http://libris.kb.se/resource/bib/219553" (URL, which is a Charmap when using either CHR or ICU). If we query Zebra with "http://libris.kb.se/resource/bib/219553", it will normalize the query the same way that it normalized "http://libris.kb.se/resource/bib/219553" when it was originally indexing it, and we'll get a match. Of course, we can't necessarily stop using C4::Matcher::_normalize() as it's the default behaviour. Many people may count on that _normalize() without even knowing it... even if it's potentially working badly. I think what I want to do is create a new normalizer which does nothing, and call it "none" or "raw". That way, I'm passing to Zebra the same thing that it's seen before, and it will normalize it exactly the same way and the likelihood of an accurate match increases considerably. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Of course, what I've said isn't 100% true. There's one more obstacle between C4::Matcher::get_matches and Zebra's normalized indexes. And that's C4::Search::SimpleSearch(), which uses the s/:/=/g regex before sending the query. While that's trivial for Zebra's CHR and ICU indexing for words and phrases, since they just strip out punctuation anyway, it's a problem for Zebra's URL Charmap, which doesn't normalize URLs. So you'll get a failed match in that case. So I think a pure match would have to be done without any normalization before the query gets to Zebra. That should be easy enough to implement in C4::Matcher when it uses C4::Search::SimpleSearch(). Because it's a simple search, it shouldn't be normalizing the query anyway. I'll just add a flag so that you can do an unnormalized search. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Create additional |Prevent normalization |normalizers for Record |during matching/import |Matching Rules |process -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- This is turning out to be harder than expected... In C4::Matcher::get_matches, it creates a query like "qualifier,qualifier:value" for QP, or "qualifier,qualifier=value" for non-QP. That's OK... but in C4::Search::SimpleSearch, QP gets turned off by "qualifier,qualifier" so it becomes a non-QP query... but it's created with a ":" instead of a "=". This matters because we need a "=" instead of a ":" for the non-QP query... and we can't wholesale change ":" to "=" because of the URL syntax required by the URL register... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=15555 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- I've solved this one... but now I need to write up an easy test plan... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- Created attachment 46848 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=46848&action=edit Bug 15541 - Prevent normalization during matching/import process This patch allows you to use the "qualifier,qualifier" syntax in the Record Matching Rules "Search Index" when using the QueryParser. While QueryParser doesn't support this syntax, it will now fallback correctly to non-QueryParser functionality. Without the patch, your search will just fail silently. This patch also adds a "skip_normalize" flag to C4::Search::SimpleSearch(), and sets the flag during C4::Matcher::get_matches. This prevents the s/:/=/g and s/=/:/g normalization which happens in a heavy-handed way to provide correct query syntax. However, get_matches() already uses the correct syntax, so this normalization is unneeded. The normalization also mangles URLs which causes failures when using the "url" (ie "u") register in Zebra (see bug 15555). This patch also creates "raw" and "none" normalizers for the Record Matching Rules, which prevents the stripping of spaces and punctuation by Koha prior to sending queries to Zebra. This is important for a number of reasons. First and foremost, Zebra does normalization better than Koha. ICU and CHR normalize strings differently, so it's better not to try to outsmart Zebra with pre-normalization of punctuation and spaces, as it will lead to search problems. Second, when using the "u" register in Zebra, you don't want to normalize the value, since it's stored "as is" in the Zebra database. Normalization causes search failures. _TEST PLAN_ 1) Apply patches from http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 2) Create a record in Koha with a 024 $a http://koha-community.org/test $2 uri 3) Do a full re-index of Zebra 4) Create a Record Matching Rule in Koha with the following details: Description: Test Match threshold: 100 Record type: Bibliographic Match point 1: Search index: id-other,st-urx Score: 100 Tag: 024 Subfields: a Normalization rule: raw 5) Download your record from Koha as a .mrc file (ie isomarc, binary marc, etc) 6) Go to "Stage MARC records for import" 7) Upload your .marc file. 8) Change your "Record matching rule" to "Test" 9) Click Stage for import 10) It should say "1 records with at least one match in catalogu per matching rule "Test". -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Academy Status|NEW |Needs Signoff Depends on| |15555 Referenced Bugs: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 [Bug 15555] Index 024$a into Identifier-other:u url register when source $2 is uri -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Marked this with the "Academy" keyword, although it might be a bit advanced... More than happy to walk people through the Zebra stuff relating to this bug though. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 Bouzid <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #46848|0 |1 is obsolete| | --- Comment #7 from Bouzid <bouzid.fergani@inlibro.com> --- Created attachment 47022 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=47022&action=edit Bug 15541 - Prevent normalization during matching/import process This patch allows you to use the "qualifier,qualifier" syntax in the Record Matching Rules "Search Index" when using the QueryParser. While QueryParser doesn't support this syntax, it will now fallback correctly to non-QueryParser functionality. Without the patch, your search will just fail silently. This patch also adds a "skip_normalize" flag to C4::Search::SimpleSearch(), and sets the flag during C4::Matcher::get_matches. This prevents the s/:/=/g and s/=/:/g normalization which happens in a heavy-handed way to provide correct query syntax. However, get_matches() already uses the correct syntax, so this normalization is unneeded. The normalization also mangles URLs which causes failures when using the "url" (ie "u") register in Zebra (see bug 15555). This patch also creates "raw" and "none" normalizers for the Record Matching Rules, which prevents the stripping of spaces and punctuation by Koha prior to sending queries to Zebra. This is important for a number of reasons. First and foremost, Zebra does normalization better than Koha. ICU and CHR normalize strings differently, so it's better not to try to outsmart Zebra with pre-normalization of punctuation and spaces, as it will lead to search problems. Second, when using the "u" register in Zebra, you don't want to normalize the value, since it's stored "as is" in the Zebra database. Normalization causes search failures. _TEST PLAN_ 1) Apply patches from http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 2) Create a record in Koha with a 024 $a http://koha-community.org/test $2 uri 3) Do a full re-index of Zebra 4) Create a Record Matching Rule in Koha with the following details: Description: Test Match threshold: 100 Record type: Bibliographic Match point 1: Search index: id-other,st-urx Score: 100 Tag: 024 Subfields: a Normalization rule: raw 5) Download your record from Koha as a .mrc file (ie isomarc, binary marc, etc) 6) Go to "Stage MARC records for import" 7) Upload your .marc file. 8) Change your "Record matching rule" to "Test" 9) Click Stage for import 10) It should say "1 records with at least one match in catalogu per matching rule "Test". Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 Bouzid <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bouzid.fergani@inlibro.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 Bouzid <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 Bouzid <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |philippe.blouin@inlibro.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Thanks Bouzid! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 --- Comment #9 from Bouzid <bouzid.fergani@inlibro.com> --- I test with: File empty File contain 1 marc record File contain several marc record -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org