[Bug 36261] New: Calculating OpacHiddenItems is too slow in OPAC search
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 Bug ID: 36261 Summary: Calculating OpacHiddenItems is too slow in OPAC search Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org Calculating items and records hidden by OpacHiddenItems it slow. Too slow. Fortunately, I've come up with an optimization which seems to effectively eliminate the overhead added by OpacHiddenItems. Basically, instead of running 1 query per item, I run 1 query per bib to get all hidden items. It's pretty straightforward, but I'd like to battle-test it a bit more. I'm also writing this optimization for an older version of Koha, and I still have more optimizing I want to do, so I'm not going to post my patch just yet... It's coming though! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Architecture, internals, |Searching |and plumbing | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36239 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 Andreas Jonsson <andreas.jonsson@kreablo.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andreas.jonsson@kreablo.se --- Comment #1 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- Ideally we should resolve this at indexing time so we get the information directly from the search index. Also the systempreference OpacHiddenItemsHidesRecord could be implemented properly, as it now breaks the result count, and theoretically you could get a blank search result page with a positive result count, and visible results at page two. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- (In reply to Andreas Jonsson from comment #1)
Ideally we should resolve this at indexing time so we get the information directly from the search index.
You can't do it at indexing time, since OpacHiddenItems can be changed at any time. (The OpacHiddenItems feature in general is a bad idea imho.)
Also the systempreference OpacHiddenItemsHidesRecord could be implemented properly, as it now breaks the result count, and theoretically you could get a blank search result page with a positive result count, and visible results at page two.
I don't think this is possible since the calculation is too complex. Personally, I'd rather deprecate OpacHiddenItemsHidesRecord and just require OpacSuppression to be used for any record that must be completely hidden. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- We use this feature a lot of the libraries and it always depends on the use case. If you just want to hide something like on order items it works certainly well enough. But maybe it should come with a warning. I'd be against deprecating it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #4 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- This is a setting that is rarely changed, I think it is justified to require a reindexing for changes to take effect. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 Hugo Agud <hagud@orex.es> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hagud@orex.es --- Comment #5 from Hugo Agud <hagud@orex.es> --- for us it is also widely used... I should not remove this feature.... on the other hand improve speed or reindex is not a big issue for us.. I do not know big catalogues it takes a lot. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to Andreas Jonsson from comment #4)
This is a setting that is rarely changed, I think it is justified to require a reindexing for changes to take effect.
That's a good point. I suppose we do have a number of other settings that require reindexes/restarts. We could add that to the explanation. That said, I'm still not sure it's possible at indexing time, especially since the search is used by both the OPAC and the staff interface. You couldn't just suppress entire records or items at indexing time without causing systemic problems. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #3)
We use this feature a lot of the libraries and it always depends on the use case. If you just want to hide something like on order items it works certainly well enough. But maybe it should come with a warning. I'd be against deprecating it.
(In reply to Hugo Agud from comment #5)
for us it is also widely used... I should not remove this feature.... on the other hand improve speed or reindex is not a big issue for us.. I do not know big catalogues it takes a lot.
It's one of those features that "kinda sorta works - good enough". Widely used but also widely problematic... By the way, I wasn't talking about deprecating OpacHiddenItems but rather OpacHiddenItemsHidesRecord. Although in theory if you did calculate OpacHiddenItems at indexing time you could turn on OpacSuppression if OpacHiddenItemsHidesRecord is enabled. That would fix some sins... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- For what it's worth, I've got my performance optimization running in 23.11 production. It doesn't solve any of the problems of OpacHiddenItemsHidesRecord or facets for hidden items appearing, but it makes a bad situation less bad in terms of performance overheads... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- Apologies that I haven't had time to upstream this. Hopefully I'll have time at some point... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #10 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- (In reply to David Cook from comment #6)
(In reply to Andreas Jonsson from comment #4)
This is a setting that is rarely changed, I think it is justified to require a reindexing for changes to take effect.
That's a good point. I suppose we do have a number of other settings that require reindexes/restarts. We could add that to the explanation.
That said, I'm still not sure it's possible at indexing time, especially since the search is used by both the OPAC and the staff interface. You couldn't just suppress entire records or items at indexing time without causing systemic problems.
When it comes to filtering items, I'm thinking that, in the search index, there should be a record level field for "no opac visible items" and for each item there should be a flag "hidden in opac". So, you can filter the search on the record level so you only get records that have at least one opac visible item. And then the information about which items that should be excluded in the result view is available directly from the search index. No additional processing would be needed at search time. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- (In reply to Andreas Jonsson from comment #10)
When it comes to filtering items, I'm thinking that, in the search index, there should be a record level field for "no opac visible items" and for each item there should be a flag "hidden in opac".
So, you can filter the search on the record level so you only get records that have at least one opac visible item. And then the information about which items that should be excluded in the result view is available directly from the search index. No additional processing would be needed at search time.
Yeah, it's an idea, and increasingly I'm thinking it could be the way to go. We'd just need to add some explanation in the syspref description for OpacHiddenItems that they'll need to rebuild the hidden statuses if/when they make changes. (We do that with a number of other settings in Koha, so there's already a precedent, so why not, right?) I think we'd need to build a new script for that, or maybe touch_all_items.pl would be enough. I guess we'd find out during testing... -- So yeah... I reckon someone should give it a go. I probably won't do the coding for it, but I'd be happy to test/QA it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Andreas Jonsson from comment #10)
When it comes to filtering items, I'm thinking that, in the search index, there should be a record level field for "no opac visible items" and for each item there should be a flag "hidden in opac".
So, you can filter the search on the record level so you only get records that have at least one opac visible item. And then the information about which items that should be excluded in the result view is available directly from the search index. No additional processing would be needed at search time.
Actually, that might still not be true. If a bib record isn't hidden, then bug 14007 would probably still be a problem. (Although only while we're using Zebra. In the future with Elasticsearch, we might be able to tweak the document so that item data isn't returned under some conditions...) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=31755 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 --- Comment #13 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- Created attachment 189696 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189696&action=edit Bug 36261: add a new column 'other' to items and deleteditems table Test plan: - apply patch - Add a new koha/marc link between 952$z and the "items.other" column. - Set the syspref OpacHiddenItems to "other: [1]" - in the default framework change the 952$z and put authorised value "YES_NO". - modify an item and set the 952$z with value "YES". - run batchRebuildItemsTable.pl. - check the item is hidden from Opac. - if the record only has one item and "OpacHiddenItemHidesRecord" the record should not be visible either. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arthur.suzuki@biblibre.com --- Comment #14 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- Hello, We've partially fixed this issue by adding a column to the items and deleteditems table. This way the "hidden to opac" can be built from the marc data beforehands. Also we had a issue because we wanted to have the information in a specific subfield of the item (rather than a combination of factors) and we couldn't map it to any column of the items table. Adding the column may also serve other purpose depending on the library configuration. Arthur -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|dcook@prosentient.com.au |arthur.suzuki@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36261 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Status|Needs Signoff |Failed QA --- Comment #15 from David Nind <david@davidnind.com> --- When I go to edit a record after making the changes to the Koha to MARC mapping and frameworks, I get an error message: Something went wrong when loading the table. 500: Internal Server Error. Properties not allowed: other. Then the table sits on "processing". Note: I don't understand this area of Koha very well (Koha to MARC mapping), so maybe I'm missing some steps? Testing notes so far: 1. Apply the patch 2. Update the database: updatedatabase 3. Step"Add a new koha/marc link between 952$z and the "items.other" column.": - Go to Koha administration > Catalog > Koha to MARC mapping - Find the items.other field on the page - Click + Add - Enter: 952,z 4. Step "in the default framework change...": I updated the default framework and the Books framework 5. Step "run batchRebuildItemsTable.pl": I ran both misc/batchRebuildBiblioTables.pl -c and misc/batchRebuildItemsTables.pl -c Other notes: 1. In the default framework for Koha, MARC21 system, 952$z is already defined as a public note and mapped to items.itemnotes 2. How would you make this work for an existing installation? -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org