https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 Paul Derscheid <paul.derscheid@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.derscheid@lmscloud.de -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #1 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- We need to implement normalization for authorities to provide the correct forms used for linking, and also implement linking We need to implement 'See from' inclusion in records Browse functionality would be nice, or 'scan' as in zebra We should introduce a new pluggable/flexible template for processing search results that we can then port ES to, rather than continuing to build on the original Zebra only base and adding exceptions (kill C4::Search) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens (kidclamp) from comment #1)

We need to implement normalization for authorities to provide the correct forms used for linking, and also implement linking

Normalization is such a loaded word, eh? This talks about normalizing authorities but it's for normalizing them in the database schema sense not the search sense: https://github.com/HKS3/koha-normalize-marc2db/blob/main/Koha/Plugin/HKS3/No... That said, I have a memory from Kohacon of Mark talking about this... Looking at https://github.com/HKS3/koha-nm2db-keyword-search/blob/main/Koha/Plugin/HKS3... I can see the addition of a FULLTEXT INDEX and the use of MATCH(), but I guess that's also in a search sense rather than a MARC search sense... Because with Zebra/Elastic it's not just that we're indexing the MARC data... but we index it in particular ways I think. And speaking of authorities... yeah how would this database search handle linked bibs? By searching all subfield $9? Very interesting -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 --- Comment #6 from Mark Hofstetter <mark@hofstetter.at> --- Created attachment 204516 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204516&action=edit Bug 42164: Allow plugins to provide search engines This patch adds the minimum core integration needed to test alternative search engines without putting a proof-of-concept implementation directly into Koha core. Enabled plugins can register Search, QueryBuilder, and Indexer classes through a koha_search_engine method. The SearchEngine system preference then offers the built-in Zebra and Elasticsearch engines together with the engines registered by plugins. The database-backed reference implementation is: https://github.com/HKS3/koha-nm2db-keyword-search It currently depends on normalized MARC tables provided by: https://github.com/HKS3/koha-normalize-marc2db Database architecture and security ---------------------------------- A database-backed search engine also exposes a broader architectural issue. This concerns the MariaDB service account, not a Koha staff or patron account. A standard Koha instance uses one database identity from koha-conf.xml for its normal DBI/DBIx connections, and the Debian installation grants that identity ALL PRIVILEGES on the instance database. The OPAC, staff interface, background workers, command-line tools, upgrades, reports, and plugins consequently use the same highly privileged database identity. The same schema also contains primary records, configuration, denormalized/generated data, transient state, and plugin-owned tables. Koha already keeps some caches and indexes outside MariaDB through Memcached and Zebra/Elasticsearch, so this is not a claim that all cache data is stored in the main database. The limitation is the lack of a general way to assign different connections, schemas, credentials, and data lifecycles to different components. Additional database identities could improve security through least privilege: * An OPAC identity could have SELECT access to public catalogue and availability data, no access to unrelated administrative or acquisition tables, and narrowly scoped write permissions for enabled patron operations such as holds, renewals, suggestions, tags, and account updates. It would not have DDL privileges. It cannot be completely read-only because several OPAC features legitimately write data. * A reports identity could be read-only, ideally against allowlisted views or a read replica so that sensitive patron, authentication, and financial columns can be excluded or masked. * A staff runtime identity could have the DML rights needed for normal staff work but no CREATE, ALTER, or DROP privileges. * A migration identity could hold DDL rights and only be available during installation and upgrades. * Search engines and plugins could use component-specific identities and, where appropriate, separate schemas containing rebuildable data. This would reduce the impact of a compromised OPAC, plugin, or report: the compromised component would not automatically gain access to every table or be able to change the schema. It would also improve auditability and credential rotation. Database permissions would supplement, not replace, Koha's application authorization. The attached patch does not introduce multiple database connections. It makes the search layer extensible so the database-search approach can be evaluated first. Named connections and documented privilege profiles would be a possible follow-up, while retaining the current single-account setup as the compatibility default. Test plan --------- 1. Apply the attached patch to current master and enable Koha plugins. 2. Run: prove t/Koha/SearchEngine.t 3. Confirm Administration > System preferences > SearchEngine still contains Zebra and Elasticsearch. 4. Install and enable the latest NormalizeMARC2DB plugin, then use "Run tool" once to normalize the existing bibliographic and authority records. 5. Install and enable the NM2DBKeywordSearch reference plugin. 6. Return to the SearchEngine preference. Confirm DBSearch is now offered and select it. 7. Search in both the OPAC and staff catalogue. Verify result pages, facets, refinements, year limits, and pagination. 8. From a Koha shell, run the reference comparison test: prove -v /path/to/koha-nm2db-keyword-search/t/nm2db_vs_koha_search.t 9. Disable the NM2DBKeywordSearch plugin and confirm DBSearch is no longer offered as a preference choice. Switch back to Zebra or Elasticsearch before testing normal searches. The reference integration test compares the database-backed result sets with Koha's normal keyword search for representative terms and also checks facets, search-within-results, availability, publication-year limits, and pagination. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 --- Comment #7 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- Created attachment 208625 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208625&action=edit Bug 42164: Allow plugins to provide search engines Allow enabled plugins to register search, query builder, and indexer classes through a koha_search_engine method. Make the SearchEngine system preference use the built-in engines plus registrations returned by enabled plugins. This keeps Zebra and Elasticsearch as the built-in defaults while allowing a database-backed search implementation to be developed and tested outside Koha core. The NM2DB Keyword Search plugin is a reference implementation: https://github.com/HKS3/koha-nm2db-keyword-search It uses the normalized MARC tables provided by: https://github.com/HKS3/koha-normalize-marc2db Test plan: 1. Apply the patch, enable plugins then run "restart_all" in KTD. 2. Run `prove t/Koha/SearchEngine.t`. 3. Verify Zebra and Elasticsearch remain available in the SearchEngine system preference. 4. Install and enable NormalizeMARC2DB, then run its tool once to normalize the existing records. 5. Install and enable NM2DBKeywordSearch, then run the plugin's tool and click "Refresh search caches". 6. Verify DBSearch appears in the SearchEngine preference and select it. 7. Run catalogue searches in the OPAC and staff interface. 8. Run the reference plugin integration test: `prove -v /path/to/koha-nm2db-keyword-search/t/nm2db_vs_koha_search.t`. AI Assistance: OpenAI Codex substantially assisted with implementation, debugging, tests, and documentation. Human author Mark Hofstetter directed the work, reviewed and approved all changes, and verified the test results. Assisted-by: GPT-5 (OpenAI) Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- (In reply to Arthur Suzuki from comment #8)

Got it working, I'm quite excited about the future modularity that could bring. The MariaDB search actually provides better results than Zebra! When trying with "中国云图" : -> with Zebra I get 21 results, most of them not related (with arabic characters?!)

Are you using ICU or CHR indexing? Zebra should've been able to handle that search.

-> with DBSearch : only one record, the "cloud atlas of the middle empire" which is the litteral translation of "中国云图" -> better correctness imo.

That's cool -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 --- Comment #10 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- This patch looks fine on itself but "In discussion" is never far away ;) Should we open the door for external plugins on such a (still) essential element of indexing our system? The work done for the two plugins needed for DBSearch is impressive. And it contains a lot of code. Should that code not actually better be moved to our codebase (at some point) when this would be kind of the (only) budget alternative for Elastic? This would make maintaining the plugin for future versions a community effort. So something like a core plugin? Hoping for some more feedback from other developers. Thanks for the work. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 --- Comment #12 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- (In reply to Mark Hofstetter from comment #11) That's a broader topic but YES: Also a big fan of the "core plugin" idea. I like the idea of modularity in general plus plugin now a days makes it easier to experiment with something before reaching community. Some of the modules could even be plugins with their own configuration rather than adding tons of syspref in Koha (did I said covers?) The ERM module already have some parts which ARE plugins as demonstrated at latest KohaCon by KIT. Totally agree with Mark, first step we can get this one in which doesn't hurt much, then make a separate patch for core plugins. (which should maybe just be handled as normal plugin but put in Koha's repository within a "plugin" directory?) as a good example : https://github.com/wled/WLED/blob/main/usermods/readme.md -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 --- Comment #14 from Mark Hofstetter <mark@hofstetter.at> --- OK as there seems to consensus that the generell direction is worth to persue, we would need QA. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42164 Frédéric Demians <f.demians@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |f.demians@tamil.fr -- You are receiving this mail because: You are watching all bug changes.