[Bug 39541] New: Do we need to store the whole MARC record in Elasticsearch?
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39541 Bug ID: 39541 Summary: Do we need to store the whole MARC record in Elasticsearch? Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: domm@plix.at QA Contact: testopia@bugs.koha-community.org Currently the whole MARC Record is stored in the ElasticSearch Record. At the Koha Hackfest 2025 I learned that one reason for this is to be able to search the whole record (Bug 22258). Another reason might be to avoid hitting the DB when displaying search results. But: * We sometime encounter problems with very large records (due to limits with UNIMARC) * The code to store the whole record is rather convoluted (involes encoding/decoding the record, checking for errors, falling back to other encoding methods..) * It bloats the ES index (if it's not needed) So I will investigate: * would displaying search results work without the whole MARC record? * aren't we hitting the DB anyway during displaying search results * in OPAC * in Staff * in API If it seems easy to optionally not add the whole record to the index, I would suggest * Adding a new Option "Don't store" to syspref ElasticsearchMARCFormat * If it is set, skip adding the whole MARC -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39541 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |domm@plix.at -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39541 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #1 from David Cook <dcook@prosentient.com.au> --- I find this idea interesting as well. Another benefit is that it would help us to separate away from MARC in the future. We can index anything into our search engine, do a consistent search, and then do display either using the search engine result or using whatever makes sense on a per record basis. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39541 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- This isn't a topic that I would urgently pursue, but if someone (ie you Thomas) plan to work on it, I'd be happy to do test/QA on it. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39541 --- Comment #3 from Thomas Klausner <domm@plix.at> --- Yesterday at the hackfest I learned that z3259 / SRU also uses the MARC data from the search index. (when using Koha as a z3950/sru server). So that's one more use case where the whole MARC record needs to be stored in the search index. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39541 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Thomas Klausner from comment #3)
Yesterday at the hackfest I learned that z3259 / SRU also uses the MARC data from the search index. (when using Koha as a z3950/sru server). So that's one more use case where the whole MARC record needs to be stored in the search index.
You mean with the z3950_responder.pl? I mean technically it wouldn't have to but that's good to know. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org