[Bug 15555] New: Index 024$a into Identifier-other:u url register when source $2 is uri
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 Bug ID: 15555 Summary: Index 024$a into Identifier-other:u url register when source $2 is uri Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Z39.50 / SRU / OpenSearch Servers Assignee: gmcharlt@gmail.com Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl Currently, 024$a is indexed into Identifier-other:w, even when it is a URI (e.g. http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14217) This causes problems because the "w" index type replaces punctuation with spaces, and tokenizes on spaces, so that the URI is decomposed into a series of values which are indexed separately. This is definitely not what you want when indexing a 024$a when it is a URI. For example: The url "http://libris.kb.se/resource/bib/219553" becomes the following: <index name="Identifier-other" type="w" seq="28">@^</index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="29">http</index> <index name="Identifier-other" type="w" seq="30">libris</index> <index name="Identifier-other" type="w" seq="31">kb</index> <index name="Identifier-other" type="w" seq="32">se</index> <index name="Identifier-other" type="w" seq="33">resource</index> <index name="Identifier-other" type="w" seq="34">bib</index> <index name="Identifier-other" type="w" seq="35">219553</index> Fortunately, the 024$2 subfield value tells us the source of the identifier, and "uri" is one of the valid options. So, when we have a 024$2=uri, we can index the 024$a using the "url" index type. (I'm also planning to index into the "phrase" index type for all 024$a as it performs the normalization but it doesn't tokenize based on the spaces, so this normal form may still be of use for urls and other identifiers that rely on punctuation for meaning.) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- Created attachment 46512 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=46512&action=edit Bug 15555 - Index 024$a into Identifier-other:u url register when source $2 is uri This patch indexes 024$a into the "phrase" index type, and the "url" index type, if the 024$2 equals "uri". TEST PLAN 1) Apply the patch. 1b) If you're using a gitified Koha or a git install, you'll need to upgrade your instance or copy your zebradb files over to /etc/koha/zebradb or your "kohadev" directory. 2) Add a 024$a with a URL like http://libris.kb.se/resource/bib/219553 to a bibliographic record 3) Re-index Zebra 4) Type "id-other,st-urx,fuzzy=http://libris.kb.se/resource/bib/219553" into the "Search the catalog" box in the Staff Client and search 5) Note that you retrieve your record NOTE: The fuzzy is required because Koha's query "parsing" functions change http:// to http=// which won't correctly match the value in the "Identifier-other:u" index. NOTE: Alternatively, you could do the following search instead: "id-other,phr=http libris kb se resource bib 219553". It would work as well by using the "Identifier-other:p" index. Advanced tester version: 4) In a terminal window, find the "koha-conf.xml" file in your "etc" directory. 5) Open "koha-conf.xml" and find <listen id="biblioserver">. Copy the URI you find there. (e.g. unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket). 6) Type "yaz-client unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket" 7) After it connects, type "base biblios" and press enter 8) Type "format xml" and press enter 9) Type "elements zebra::index" and press enter 10) Type "f id-other,st-urx=http://libris.kb.se/resource/bib/219553" and press enter 11) Note that you should have at least one result 12) Type "show 1" 13) If you scroll through the results, you should find something like the following: <index name="Identifier-other" type="w" seq="28">@^</index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="29">http</index> <index name="Identifier-other" type="w" seq="30">libris</index> <index name="Identifier-other" type="w" seq="31">kb</index> <index name="Identifier-other" type="w" seq="32">se</index> <index name="Identifier-other" type="w" seq="33">resource</index> <index name="Identifier-other" type="w" seq="34">bib</index> <index name="Identifier-other" type="w" seq="35">219553</index> <index name="Identifier-other" type="p" seq="28">http libris kb se resource bib 219553</index> <index name="Identifier-other" type="u" seq="36">http://libris.kb.se/resource/bib/219553</index> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Depends on| |14217 Referenced Bugs: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14217 [Bug 14217] Add a DOM syntax for specifying conditions on indexes -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Advanced testers: Here's the steps you need to do to upgrade your Koha's Zebra config, if you're using a git dev install: cd /home/dcook/git INSTALL_LOG="/home/dcook/koha-dev/misc/koha-install-log" cp $INSTALL_LOG backup-koha-install-log perl Makefile.PL --prev-install-log "$INSTALL_LOG" make make upgrade -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- NOTE: The output I've provided in the above comments has come from a Koha using CHR indexing. ICU indexing has different output, but the same behaviour. Here's what I see with "format xml" and "elements zebra::index": <index name="Identifier-other" type="w" seq="21"></index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="22"></index> <index name="Identifier-other" type="w" seq="23"></index> <index name="Identifier-other" type="w" seq="24"></index> <index name="Identifier-other" type="w" seq="25"></index> <index name="Identifier-other" type="p" seq="21"></index> <index name="Identifier-other" type="p" seq="22"></index> <index name="Identifier-other" type="p" seq="23"></index> <index name="Identifier-other" type="p" seq="24"></index> <index name="Identifier-other" type="p" seq="25"></index> <index name="Identifier-other" type="u" seq="26">http://libris.kb.se/resource/bib/219553</index> Here's what I see with "format xml" and "elements index": <z:index name="Identifier-other:w Identifier-other:p">http://libris.kb.se/resource/bib/219553</z:index> <z:index name="Identifier-other:u">http://libris.kb.se/resource/bib/219553</z:index> However, this output is misleading. That's basically just the output of "xsltproc biblio-zebra-indexdefs.xsl <record>". It's pre-normalization and thus essentially meaningless. ----------- Only advanced users will look at yaz-client though. That being said, there are functional differences between ICU and CHR. For instance, the following query will work in CHR but NOT in ICU: id-other,phr=http libris kb se resource bib 219553 Conversely, the following query will work in ICU but not in CHR: id-other,phr=http libriskbse resource bib 219553 That's because we've configured tokenization and normalization to work differently between the two schemes. Fun, right? ICU uses the "l" tokenize rule from http://www.indexdata.com/yaz/doc/yaz-icu.html. That means it tokenizes based on slashes, spaces, and maybe some other characters I haven't discovered yet. You can verify that with the following commands: echo "THIS IS A TEST" | yaz-icu -x -c ./etc/zebradb/etc/phrases-icu.xml echo "THIS/IS/A/TEST" | yaz-icu -x -c ./etc/zebradb/etc/phrases-icu.xml Indeed, check out the following yaz-client output when using ICU: Z> f id-other,phr=http://libris.kb.se/resource/bib/219553 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 12, setno 17 SearchResult-1: term=http cnt=12, term=libriskbse cnt=12, term=resource cnt=12, term=bib cnt=12, term=219553 cnt=12 records returned: 0 Elapsed: 0.001458 You can see the URL has been broken into 5 terms/tokens with ICU, while CHR does the following: Z> f id-other,phr=http://libris.kb.se/resource/bib/219553 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 12, setno 13 SearchResult-1: term=http cnt=12, term=libris cnt=12, term=kb cnt=12, term=se cnt=12, term=resource cnt=12, term=bib cnt=12, term=219553 cnt=12 records returned: 0 Elapsed: 0.001119 We actually have 7 terms/tokens in the case of CHR! And that means that we don't want to try to outsmart Zebra by pre-normalizing our queries. We want to query Zebra with the exact same data that it indexed, because that way the normalization will be the same! Science! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Note that the above pertains to "phrases-icu.xml" specifically. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 Héctor Eduardo Castro Avalos <hector.hecaxmmx@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 Héctor Eduardo Castro Avalos <hector.hecaxmmx@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #46512|0 |1 is obsolete| | --- Comment #5 from Héctor Eduardo Castro Avalos <hector.hecaxmmx@gmail.com> --- Created attachment 46567 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=46567&action=edit [SIGNED-OFF]Bug 15555: Index 024$a into Identifier-other:u url register when source $2 is uri This patch indexes 024$a into the "phrase" index type, and the "url" index type, if the 024$2 equals "uri". TEST PLAN 1) Apply the patch. 1b) If you're using a gitified Koha or a git install, you'll need to upgrade your instance or copy your zebradb files over to /etc/koha/zebradb or your "kohadev" directory. 2) Add a 024$a with a URL like http://libris.kb.se/resource/bib/219553 to a bibliographic record 3) Re-index Zebra 4) Type "id-other,st-urx,fuzzy=http://libris.kb.se/resource/bib/219553" into the "Search the catalog" box in the Staff Client and search 5) Note that you retrieve your record NOTE: The fuzzy is required because Koha's query "parsing" functions change http:// to http=// which won't correctly match the value in the "Identifier-other:u" index. NOTE: Alternatively, you could do the following search instead: "id-other,phr=http libris kb se resource bib 219553". It would work as well by using the "Identifier-other:p" index. Advanced tester version: 4) In a terminal window, find the "koha-conf.xml" file in your "etc" directory. 5) Open "koha-conf.xml" and find <listen id="biblioserver">. Copy the URI you find there. (e.g. unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket). 6) Type "yaz-client unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket" 7) After it connects, type "base biblios" and press enter 8) Type "format xml" and press enter 9) Type "elements zebra::index" and press enter 10) Type "f id-other,st-urx=http://libris.kb.se/resource/bib/219553" and press enter 11) Note that you should have at least one result 12) Type "show 1" 13) If you scroll through the results, you should find something like the following: <index name="Identifier-other" type="w" seq="28">@^</index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="29">http</index> <index name="Identifier-other" type="w" seq="30">libris</index> <index name="Identifier-other" type="w" seq="31">kb</index> <index name="Identifier-other" type="w" seq="32">se</index> <index name="Identifier-other" type="w" seq="33">resource</index> <index name="Identifier-other" type="w" seq="34">bib</index> <index name="Identifier-other" type="w" seq="35">219553</index> <index name="Identifier-other" type="p" seq="28">http libris kb se resource bib 219553</index> <index name="Identifier-other" type="u" seq="36">http://libris.kb.se/resource/bib/219553</index> Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com> Works as advertised the record is retrieved -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 Héctor Eduardo Castro Avalos <hector.hecaxmmx@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hector.hecaxmmx@gmail.com, | |nicole@bywatersolutions.com --- Comment #6 from Héctor Eduardo Castro Avalos <hector.hecaxmmx@gmail.com> --- Because of your NOTE I think IMHO this needs to go in Manual. Add here Documentation Manager -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 Nicole C. Engard <nengard@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nengard@gmail.com --- Comment #7 from Nicole C. Engard <nengard@gmail.com> --- Where do we think this should go in the manual? We don't really get this nitty gritting in the searching section. Nicole -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Yeah, that might be tough to add to the manual. Prior to this patch, only the 952$u had an index with a "u" (ie url) register, and that's only accessible using "st-urx" as a CCL qualifier. Actually, in the future, I'd like to add the 856$u to the "url:u" index as well. I suppose that might be added to the manual... saying that "url,st-urx" will let you look up URLs in the 952$u and 856$u. I've had some people ask in the past about easy ways to find records with URLs and that would do it... But yeah. Not sure about the manual. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- Actually, regarding the manual, I'm tempted to personally write up an appendix regarding searching in Koha using CCL... but I'm not sure when that's going to happen. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- It would also be a bit advanced, as I would need to get into the stuff about the "p", "n", "w", and "u" registers for it to make complete sense. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=15541 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15555 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |15541 Referenced Bugs: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15541 [Bug 15541] Prevent normalization during matching/import process -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org