[Bug 12872] New: Exclude limited number of fields from Zebra DOM indexing
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Bug ID: 12872 Summary: Exclude limited number of fields from Zebra DOM indexing Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: gmcharlt@gmail.com Reporter: m.de.rooy@rijksmuseum.nl QA Contact: testopia@bugs.koha-community.org Here is another approach for excluding a limited number of fields from Zebra DOM indexing. (Check the see also reports too.) I rename index_all to index_most and add a few lines to that named template. This will not win beauty prices, but imo it is easier than adding any or anywhere to a large number of fields. What do you think? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #1 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 31366 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31366&action=edit Bug 12872: Exclude limited number of fields from Zebra DOM indexing This is a pragmatic approach to exclude some fields from the Any index. The index_all routine in koha-indexdefs-to-zebra.xsl is renamed to index_most and includes a choose-when construct to exclude some fields from indexing. This seems a simpler approach than removing this routine and adding Any almost anywhere :) Test plan: Put some uncommon string like jh234gj23 into 952$e somewhere. Copy the xsl file to your Zebra indexing folders. Run the xsl to produce a new biblio-zebra-indexdefs.xsl. Restart Zebra and reindex. Check if you can still find the uncommon word. Now uncomment the line in koha-indexdefs-to-zebra.xsl. Rerun the xsl, restart and reindex. You should not find the uncommon word in the added 952$e. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=11291 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=8962 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff --- Comment #2 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Please signoff if this patch does what it promises in your setup, or please give suggestions how to handle this more elegantly. Thanks. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #31366|0 |1 is obsolete| | --- Comment #3 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 31367 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31367&action=edit Bug 12872: Exclude limited number of fields from Zebra DOM indexing This is a pragmatic approach to exclude some fields from the Any index. The index_all routine in koha-indexdefs-to-zebra.xsl is renamed to index_most and includes a choose-when construct to exclude some fields from indexing. This seems a simpler approach than removing this routine and adding Any almost anywhere :) Test plan: Put some uncommon string like jh234gj23 into 952$e somewhere. Copy the xsl file to your Zebra indexing folders. Run the xsl to produce a new biblio-zebra-indexdefs.xsl. Restart Zebra and reindex. Check if you can still find the uncommon word. Now uncomment the line in koha-indexdefs-to-zebra.xsl. Rerun the xsl, restart and reindex. You should not find the uncommon word in the added 952$e. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #4 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Oops, still the wrong version. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #31367|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #5 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 31368 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31368&action=edit Bug 12872: Exclude limited number of fields from Zebra DOM indexing This is a pragmatic approach to exclude some fields from the Any index. The index_all routine in koha-indexdefs-to-zebra.xsl is renamed to index_most and includes a choose-when construct to exclude some fields from indexing. This seems a simpler approach than removing this routine and adding Any almost anywhere :) Test plan: Put some uncommon string like jh234gj23 into 952$e somewhere. Copy the xsl file to your Zebra indexing folders. Run the xsl to produce a new biblio-zebra-indexdefs.xsl. Restart Zebra and reindex. Check if you can still find the uncommon word. Now uncomment the line in koha-indexdefs-to-zebra.xsl. Rerun the xsl, restart and reindex. You should not find the uncommon word in the added 952$e. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #31368|0 |1 is obsolete| | --- Comment #6 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Comment on attachment 31368 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31368 Bug 12872: Exclude limited number of fields from Zebra DOM indexing This approach would work, but I would rather opt for something different. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Exclude limited number of |Exclude limited number of |fields from Zebra DOM |fields from Zebra indexing |indexing | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #7 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- On second thought, I am widening the scope of this report. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Exclude limited number of |Exclude some fields from |fields from Zebra indexing |Zebra indexing -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #8 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 31527 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31527&action=edit Bug 12872: Exclude limited number of fields from Zebra indexing This patch adds the use of preference ZebraExclude to rebuild_zebra.pl. If that pref does not exist or is empty, behavior is unchanged. The pref allows you to exclude fields, specific subfields, and even field groups like 9XX from passing them to Zebra. The magic field XXX allows you to skip a whole record while indexing; this is of course only practical with a preceding specific condition. The pref may contain multiple lines with their own conditions. A line may start with a regex like /regex/:fieldlist. The regex is applied to the marcxml. If it matches, the field excludes are processed. Test plan: The plan includes a few rebuilds. It may be practical to run this test on a smaller test database. [1a] Edit ZebraExclude syspref. Set it to 590, 952e [1b] Edit a few records/items for these fields. [1c] Rebuild the zebra index. [1d] Check that info in 590 and 952e cannot be found in Zebra. [2a] Edit ZebraExclude syspref. Set it to /_skip_/: XXX [2b] Enter the _skip_ string into a few biblios (not in the same field). [2c] Rebuild the zebra index. [2d] Check that you cannot find these records in Zebra. [3a] Edit ZebraExclude syspref. Set it to 9XX [3b] Edit a few 9XX fields including some items (952). [3c] Rebuild the zebra index. [3d] Check that you cannot find 9XX info in Zebra. [4 ] Enter some other value in the pref. Edit, rebuild and check. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #9 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 31528 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31528&action=edit Bug 12872: Unit tests for ZebraExclude preference Test plan: Run ZebraExclude.t -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #10 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 31529 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31529&action=edit Bug 12872: Db revision for ZebraExclude pref Test plan: Run install or upgrade. Check insertion of pref into database. Check editing the value via interface. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff --- Comment #11 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Will still add some documentation in online help later. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #12 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to M. de Rooy from comment #11)
Will still add some documentation in online help later.
And still some POD in a follow-up for the module. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |paola.rossi@cineca.it --- Comment #13 from Paola Rossi <paola.rossi@cineca.it> --- I've applied against master 3.17.00.027 Two patches need trivial rebasing. I pass the patch to "Patch doesn't apply" status. ------------------------------------ Some problems about "any" index (searching by keyword index). For example if the subfields 999c/999d exists in a record. In intranet, if I searched the biblionumber (=value of the number in the 2 subfields) by keyword index (on advanced search) or entering search keywords on the simple search, I found that record even if the "ZebraExclude" syspref has been set to 999. I found that record by sn index too, even if the "ZebraExclude" syspref has been set to 999. ------------------------------------ -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #14 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Paola Rossi from comment #13)
I've applied against master 3.17.00.027
Two patches need trivial rebasing.
I pass the patch to "Patch doesn't apply" status.
------------------------------------ Some problems about "any" index (searching by keyword index).
For example if the subfields 999c/999d exists in a record. In intranet, if I searched the biblionumber (=value of the number in the 2 subfields) by keyword index (on advanced search) or entering search keywords on the simple search, I found that record even if the "ZebraExclude" syspref has been set to 999. I found that record by sn index too, even if the "ZebraExclude" syspref has been set to 999. ------------------------------------
Thanks for testing! The thing that you mentio about field 999 is by design. Please have a look at sub load_zebra_exclude. It uses GetMarcFromKohaField to get the tag that Koha uses to store biblionumber and biblioitemnumber. If that is indeed 999 (as it is by default), this tag will not be excluded since Zebra needs that information somewhere. I would not recommend to use the same tag where we internally store biblionumber for other data. So I did not see this as a problem. Could you retry with another field? Do not forget to reindex. Will rebase the last patch. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #31529|0 |1 is obsolete| | --- Comment #15 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 32578 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32578&action=edit Bug 12872: Db revision for ZebraExclude pref Test plan: Run install or upgrade. Check insertion of pref into database. Check editing the value via interface. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #31527|0 |1 is obsolete| | --- Comment #16 from Paola Rossi <paola.rossi@cineca.it> --- Created attachment 32584 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32584&action=edit Bug 12872: Exclude limited number of fields from Zebra indexing I've applied the patches against master 3.17.00.029 The case about 999 was the only problem I tried. Thanks, M. de Rooy, for the answer in comment 14. So everything is OK. And I pass the patch to "Signed Off" status. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #31528|0 |1 is obsolete| | --- Comment #17 from Paola Rossi <paola.rossi@cineca.it> --- Created attachment 32585 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32585&action=edit Bug 12872: Unit tests for ZebraExclude preference -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32578|0 |1 is obsolete| | --- Comment #18 from Paola Rossi <paola.rossi@cineca.it> --- Created attachment 32586 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32586&action=edit Bug 12872: Db revision for ZebraExclude pref -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32584|0 |1 is obsolete| | --- Comment #19 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 32892 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32892&action=edit Bug 12872: Exclude limited number of fields from Zebra indexing This patch adds the use of preference ZebraExclude to rebuild_zebra.pl. If that pref does not exist or is empty, behavior is unchanged. The pref allows you to exclude fields, specific subfields, and even field groups like 9XX from passing them to Zebra. The magic field XXX allows you to skip a whole record while indexing; this is of course only practical with a preceding specific condition. The pref may contain multiple lines with their own conditions. A line may start with a regex like /regex/:fieldlist. The regex is applied to the marcxml. If it matches, the field excludes are processed. Test plan: The plan includes a few rebuilds. It may be practical to run this test on a smaller test database. [1a] Edit ZebraExclude syspref. Set it to 590, 952e [1b] Edit a few records/items for these fields. [1c] Rebuild the zebra index. [1d] Check that info in 590 and 952e cannot be found in Zebra. [2a] Edit ZebraExclude syspref. Set it to /_skip_/: XXX [2b] Enter the _skip_ string into a few biblios (not in the same field). [2c] Rebuild the zebra index. [2d] Check that you cannot find these records in Zebra. [3a] Edit ZebraExclude syspref. Set it to 9XX [3b] Edit a few 9XX fields including some items (952). [3c] Rebuild the zebra index. [3d] Check that you cannot find 9XX info in Zebra. [4 ] Enter some other value in the pref. Edit, rebuild and check. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #20 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 32893 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32893&action=edit Bug 12872: Unit tests for ZebraExclude preference Test plan: Run ZebraExclude.t Signed-off-by: Paola Rossi <paola.rossi@cineca.it> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32585|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32586|0 |1 is obsolete| | --- Comment #21 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 32894 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32894&action=edit Bug 12872: Db revision for ZebraExclude pref Test plan: Run install or upgrade. Check insertion of pref into database. Check editing the value via interface. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #22 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Trivial rebase, had to fight with the mbox attachment that should have been a patch.. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |kyle@bywatersolutions.com --- Comment #23 from Kyle M Hall <kyle@bywatersolutions.com> --- This needs some end-user documentation. There really is nothing telling the user what to put in the system preference specifically. On the other hand, would it be better to use command line options for rebuild_zebra.pl for this? Considering any changes require a full rebuild, it kind of feels that way to me. Perhaps the settings could be stored in a file in etc if it's too complicated for the command line. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |olli-antti.kivilahti@jns.fi --- Comment #24 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Why is this feature needed? Cant you just change the Zebra index definitions? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 --- Comment #25 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Olli-Antti Kivilahti from comment #24)
Why is this feature needed? Cant you just change the Zebra index definitions?
Have a look at some of the obsoleted patches too. I used another approach there. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12872 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=15050 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org