https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33627 Didier Gautheron <didier.gautheron@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|OAI-PMH script |OAI-PMH script |build_oai_sets.pl is slow |build_oai_sets.pl is slow |and memory angry |and memory hungry -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33627 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=35442 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33627 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- I don't have to build sets often, so I'm probably not going to work on this in the short-term... but it would be great to improve this one day... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33627 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Another idea for build_oai_sets.pl specifically... adding a --where option. Typically, this script is only needed for the initial population of an OAI-PMH set, so we fetch everything and compare it to the mappings. However, especially with bug 37486, we can probably put together a SQL WHERE clause which helps us reduce - at the database level - the number of records to consider for the mappings. For instance, I only want bib records that have items for branch A. I don't need all biblio_metadata and deletedbiblio_metadata. I could add a WHERE clause that dramatically reduces the number of metadata records fetched from the database. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33627 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz --- Comment #5 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Improving the performance of this script would be of interest to us. Another idea Alex had was to have a table of the records in the OAI set, being added to as creation/deletion actions were taken, then to clear it out regularly. Not sure if I'm describing that right, but must be faster than trying to filter on timestamps and fetch everything altogether. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33627 --- Comment #7 from Alex Buckley <alexbuckley@catalyst.net.nz> --- (In reply to Katrin Fischer from comment #6)

(In reply to Aleisha Amohia from comment #5)

...

Improving the performance of this script would be of interest to us.

Another idea Alex had was to have a table of the records in the OAI set, being added to as creation/deletion actions were taken, then to clear it out regularly. Not sure if I'm describing that right, but must be faster than trying to filter on timestamps and fetch everything altogether.

Would this be like a "queue" table? Just wondering if it's different to OAI-PMH:AutoUpdateSets or if that could cover it.

Hey Katrin, What I'm thinking of is, either, a new database table or indeed adding a new timestamp column to the existing oai_sets_biblios table. With the purpose of optimising when Koha generates the OAI-PMH content, for harvest by other systems. --- Context: Currently the Koha/OAI/Server/ListBase.pm->GetRecords() function fetches data from a lot of different database tables to generate a list of records to display on the Koha OAI-PMH page. The UNIONS and other data processing (DISTINCT, ORDER BY, LIMIT) across multiple database tables in the SQL queries slow how long it takes to generate the record list. For libraries which include items in their OAI-PMH content, the SQL query does a UNION and fetches data from 4 database tables: oai_sets_biblios, biblio_metadata, deleteditems, items. See https://git.koha-community.org/Koha-community/Koha/src/branch/main/Koha/OAI/... --- Real world example: We have a partner library, who does include items in their OAI config. This library undertakes a lot of record imports/updates, making their VuFind harvests from Koha OAI-PMH large. This library has encountered failed OAI-PMH harvests from Koha because out timeouts on the Koha OAI-PMH side. We debugged and found the Koha/OAI/Server/ListBase.pm->GetRecords() SQL query took a very significant amount of time to complete in the database. In one case, it took 27 minutes to run, despite the database being fully configured and optimised as per community recommendations. --- Ideal solution: If the timestamp information can be stored partially, or in full, in a new oai_sets_biblios.timestamp table. Then the SQL queries in https://git.koha-community.org/Koha-community/Koha/src/branch/main/Koha/OAI/... could fetch data from fewer tables, thereby optimising how how the OAI-PMH page takes to generate. I hope this helps clarify our thinking? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.