http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |minor -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 --- Comment #1 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 31978 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31978&action=edit Bug 13022 - Hardcoded limit causes records with more than 20 items to show inaccurate statuses If a record has more than 20 items, all the items over 20 will show as available on the search results even if they are not! This is a hard coded limit in the Search module. This number should be configurable. Test Plan: 1) Create a record with more than 20 items 2) Set all the items to waiting holds or in transit 3) Search for results that will include that item 4) Note some say they are available even though they are not 5) Apply this patch 6) Run updatedatabase.pl 7) Set the new system preference MaxSearchResultsItemsPerRecordStatusCheck to a number larger than the number of items on your record 8) Re-run the search 9) Note that the hold and transit statuses for the items are now correct -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #2 from Owen Leonard <oleonard@myacpl.org> --- This is an interesting bug... I had no idea this arbitrary limitation was in the code. Note the associated FIXME: # A couple heuristics to limit how many times # we query the database for item transfer information, sacrificing # accuracy in some cases for speed; # # 1. don't query if item has one of the other statuses # 2. don't check transit status if the bib has # more than 20 items # # FIXME: to avoid having the query the database like this, and to make # the in transit status count as unavailable for search limiting, # should map transit status to record indexed in Zebra. The problem with both the old way and the patched version of things is that something is happening which the user doesn't know anything about. You can pick an arbitrarily high number for MaxSearchResultsItemsPerRecordStatusCheck (that's a mouthful!) and rest assured that no inaccurate information will be displayed or you can set it to something else and take your chances. How does the librarian administrator know what value of MaxSearchResultsItemsPerRecordStatusCheck is reasonable? I'm interested in hearing why the solution proposed in the FIXME isn't the one which has been implemented. If the Zebra solution is not a good one, I think a better solution to this might just be to add a note which tells the user to click through to the detail page for accurate status information. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #3 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to Owen Leonard from comment #2)

This is an interesting bug... I had no idea this arbitrary limitation was in the code. Note the associated FIXME:

# A couple heuristics to limit how many times # we query the database for item transfer information, sacrificing # accuracy in some cases for speed; # # 1. don't query if item has one of the other statuses # 2. don't check transit status if the bib has # more than 20 items # # FIXME: to avoid having the query the database like this, and to make # the in transit status count as unavailable for search limiting, # should map transit status to record indexed in Zebra.

The problem with both the old way and the patched version of things is that something is happening which the user doesn't know anything about. You can pick an arbitrarily high number for MaxSearchResultsItemsPerRecordStatusCheck (that's a mouthful!) and rest assured that no inaccurate information will be displayed or you can set it to something else and take your chances.

How does the librarian administrator know what value of MaxSearchResultsItemsPerRecordStatusCheck is reasonable?

I'm interested in hearing why the solution proposed in the FIXME isn't the one which has been implemented.

If the Zebra solution is not a good one, I think a better solution to this might just be to add a note which tells the user to click through to the detail page for accurate status information.

I don't think that would be a satisfactory solution for many librarians. The zebra fix is also limited. This problem isn't just inherent in the transit status, but also the lost status ( and probably the not for loan status and who knows what else ). I think the most correct solution would be to remove the limit altogether, but in the interest of those libraries that would prefer speed over accuracy, I made the limit controllable via a system preference instead. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37970|0 |1 is obsolete| | --- Comment #5 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 37971 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37971&action=edit Bug 13022 - Hardcoded limit causes records with more than 20 items to show inaccurate statuses If a record has more than 20 items, all the items over 20 will show as available on the search results even if they are not! This is a hard coded limit in the Search module. This number should be configurable. Test Plan: 1) Create a record with more than 20 items 2) Set all the items to waiting holds or in transit 3) Search for results that will include that item 4) Note some say they are available even though they are not 5) Apply this patch 6) Run updatedatabase.pl 7) Set the new system preference MaxSearchResultsItemsPerRecordStatusCheck to a number larger than the number of items on your record 8) Re-run the search 9) Note that the hold and transit statuses for the items are now correct -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37971|0 |1 is obsolete| | --- Comment #6 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- Created attachment 38066 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38066&action=edit Bug 13022 - Hardcoded limit causes records with more than 20 items to show inaccurate statuses If a record has more than 20 items, all the items over 20 will show as available on the search results even if they are not! This is a hard coded limit in the Search module. This number should be configurable. Test Plan: 1) Create a record with more than 20 items 2) Set all the items to waiting holds or in transit 3) Search for results that will include that item 4) Note some say they are available even though they are not 5) Apply this patch 6) Run updatedatabase.pl 7) Set the new system preference MaxSearchResultsItemsPerRecordStatusCheck to a number larger than the number of items on your record 8) Re-run the search 9) Note that the hold and transit statuses for the items are now correct Signed-off-by: Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #38069|0 |1 is obsolete| | --- Comment #8 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 38072 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38072&action=edit Bug 13022 [QA Followup] - Allow the pref to be unset for unlimited item status checks -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #38075|0 |1 is obsolete| | --- Comment #10 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 38076 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38076&action=edit Bug 13022 [QA Followup] - Highlight the results with inaccurate status, add explanation when mouse hovers over. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 --- Comment #12 from Jonathan Druart <jonathan.druart@biblibre.com> --- I think that the strings introduced in the last patch won't be easily/correctly translatable. + "48 items , 44 available" => There is a space before the comma. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #38437|0 |1 is obsolete| | --- Comment #14 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 38438 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38438&action=edit Bug 13022 [QA Followup] - Revert changes in whitespace back to original for items count lines -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #38066|0 |1 is obsolete| | Attachment #38072|0 |1 is obsolete| | Attachment #38076|0 |1 is obsolete| | Attachment #38438|0 |1 is obsolete| | --- Comment #16 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 38476 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38476&action=edit Bug 13022 - Hardcoded limit causes records with more than 20 items to show inaccurate statuses If a record has more than 20 items, all the items over 20 will show as available on the search results even if they are not! This is a hard coded limit in the Search module. This number should be configurable. Test Plan: 1) Create a record with more than 20 items 2) Set all the items to waiting holds or in transit 3) Search for results that will include that item 4) Note some say they are available even though they are not 5) Apply this patch 6) Run updatedatabase.pl 7) Set the new system preference MaxSearchResultsItemsPerRecordStatusCheck to a number larger than the number of items on your record 8) Re-run the search 9) Note that the hold and transit statuses for the items are now correct Signed-off-by: Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 --- Comment #18 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 38478 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38478&action=edit Bug 13022 [QA Followup] - Highlight the results with inaccurate status, add explanation when mouse hovers over. Bug 13022 [QA Followup] - Revert changes in whitespace back to original for items count lines Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com Status|Passed QA |Failed QA --- Comment #20 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I think this should be solved on indexing, no calculation on what is the status of the records/items should be done on rendering the results page. If you take a look at C4::Circulation::{AddIssue|AddReturn} you will notice that they call C4::Items::ModItem, which ultimately calls ModZebra... So we should always assume the data we retrieve from Zebra is current... and no calculation should be done outside the resultset... I belive at some point we messed up things A LOT (by adding the <items> thing in the XSLT step... but until someone rewrites all that... I think we should do the search engine-way. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 --- Comment #21 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to Tomás Cohen Arazi from comment #20)

I think this should be solved on indexing, no calculation on what is the status of the records/items should be done on rendering the results page.

If you take a look at C4::Circulation::{AddIssue|AddReturn} you will notice that they call C4::Items::ModItem, which ultimately calls ModZebra...

So we should always assume the data we retrieve from Zebra is current... and no calculation should be done outside the resultset...

I belive at some point we messed up things A LOT (by adding the <items> thing in the XSLT step... but until someone rewrites all that... I think we should do the search engine-way.

So what steps do we need to take to resolve this? It sounds like we need to create an availability index. This would also mean we need to re-index whenever other tables related to items change ( transfers and reserves at the least ). I think that this patch could be pushed in the meantime as a band-aid at least, until we can get such a system in place. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #38476|0 |1 is obsolete| | Attachment #38477|0 |1 is obsolete| | Attachment #38478|0 |1 is obsolete| | --- Comment #22 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 45957 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=45957&action=edit Bug 13022 - Hardcoded limit causes records with more than 20 items to show inaccurate statuses If a record has more than 20 items, all the items over 20 will show as available on the search results even if they are not! This is a hard coded limit in the Search module. This number should be configurable. Test Plan: 1) Create a record with more than 20 items 2) Set all the items to waiting holds or in transit 3) Search for results that will include that item 4) Note some say they are available even though they are not 5) Apply this patch 6) Run updatedatabase.pl 7) Set the new system preference MaxSearchResultsItemsPerRecordStatusCheck to a number larger than the number of items on your record 8) Re-run the search 9) Note that the hold and transit statuses for the items are now correct Signed-off-by: Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 --- Comment #24 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 45959 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=45959&action=edit Bug 13022 [QA Followup] - Highlight the results with inaccurate status, add explanation when mouse hovers over. Bug 13022 [QA Followup] - Revert changes in whitespace back to original for items count lines Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 --- Comment #26 from Kyle M Hall <kyle@bywatersolutions.com> ---

I would like to see a comment from Tomas before these patches are pushed.

I should have noted Tomas gave his blessing to have this pushed to master in a pm. I think we can all agree that the best long term solution is to index these values instead of calculating them for each search, but this is at least a decent stop-gap measure until such changes can be made. I will ask Tomas to confirm our conversation here before pushing this. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Brendan Gallagher <brendan@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |brendan@bywatersolutions.co | |m --- Comment #27 from Brendan Gallagher <brendan@bywatersolutions.com> --- (In reply to Kyle M Hall from comment #26)

...

I would like to see a comment from Tomas before these patches are pushed.

I should have noted Tomas gave his blessing to have this pushed to master in a pm. I think we can all agree that the best long term solution is to index these values instead of calculating them for each search, but this is at least a decent stop-gap measure until such changes can be made.

I will ask Tomas to confirm our conversation here before pushing this.

I am ok with this as a stop-gap. I will push this. But I am really interested to see if we can get this right before the next release is out the door. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |julian.maurice@biblibre.com --- Comment #29 from Julian Maurice <julian.maurice@biblibre.com> --- Patches pushed to 3.22.x, will be in 3.22.2 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |If a record has more than release notes| |20 items, all the items | |over 20 will show as | |available on the search | |results even if they are | |not! This is a hard coded | |limit in the Search module. | |This number should is made | |configurable with the new | |system preference | |MaxSearchResultsItemsPerRec | |ordStatusCheck. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13022 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|If a record has more than |If a record has more than release notes|20 items, all the items |20 items, all the items |over 20 will show as |over 20 will show as |available on the search |available on results even |results even if they are |if they are not! This is a |not! This is a hard coded |hard coded limit in the |limit in the Search module. |Search module. This is made |This number is made |configurable with the new |configurable with the new |system preference |system preference |MaxSearchResultsItemsPerRec |MaxSearchResultsI |ordStatusCheck. -- You are receiving this mail because: You are watching all bug changes.