https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- At the moment, it's a single threaded single Perl process that does blocking I/O, so the time it takes to query all Koha databases will degrade as you add Koha databases. While the database queries are fast, this is a potential limitation. However, I could use Mojo to do non-blocking I/O for the database queries, which would remove that potential scalability limitation. Alternatively, I could look at benchmarking the database queries to see if it really is just a hypothetical limitation. It is possible that I could get some paid work time to do this work, but I want to see what community feedback I can get before I do a POC. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- While these ideas are in the name of efficiency and scalability, there is a downside. They are based on the idea that all the relevant Koha instances are collocated on the same server as the Zebra indexing daemon. Another idea would be to have 1 process per Koha instance (with very minimal memory usage) that either polls the database or listens to RabbitMQ and then forks a child process to run rebuild_zebra.pl (or triggers an API call that does the indexing). This idea is probably the most container friendly. (Another indexing optimization would be to have rebuild_zebra.pl do its work on SSD or RAM disk, since it can perform a lot of I/O for temporarily held data which can be really slow. In the past, we've noticed huge speed improvements by having rebuild_zebra.pl work on a RAM disk.) Reference material: Reading https://stackoverflow.com/questions/9733146/tips-for-keeping-perl-memory-usa... led to many interesting thoughts on memory usage in Perl, especially the part about garbage collection not working the way I expected in Perl. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- By the way, this would be sponsored by Prosentient Systems, so this isn't a personal project. If people are happy with this idea, I will set aside a work day (or more likely parts of a few work days) and do this work. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Created attachment 114884 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114884&action=edit Bug 27267: Add memory efficient Zebra indexing daemon This patch adds an alternate Zebra indexing daemon which is much more memory efficient than the current default daemon. (For instance, zebra_indexer.pl uses 6MB of RAM vs rebuild_zebra.pl which uses about 180MB of RAM out of the box while idle.) To test: 0. Apply patch 1. koha-indexer --stop kohadev 2. vi /etc/default/koha-common 3. Update ALTERNATE_INDEXER_DAEMON to be "/kohadevbox/koha/misc/zebra_indexer.pl" 4. koha-indexer --start kohadev 5. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 6. Change 245$a to "Gone" instead of "Gairm" and save the record 7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Gairm 8. Repeat Step 7 for 5+ seconds until your search returns no results 9. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 10. Change 245$a from "Gone" to "Gairm" 11. Repeat Step 7 for 5+ seconds until your search redirects you to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29&found1=1 To confirm lower memory usage: 12. ps -efww | grep "zebra_indexer" 13. top -p <PID of zebra_indexer.pl process> 14. The top tool should show about 6.1MB usage. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27348 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #114884|0 |1 is obsolete| | --- Comment #9 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 117437 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=117437&action=edit Bug 27267: Add memory efficient Zebra indexing daemon This patch adds an alternate Zebra indexing daemon which is much more memory efficient than the current default daemon. (For instance, zebra_indexer.pl uses 6MB of RAM vs rebuild_zebra.pl which uses about 180MB of RAM out of the box while idle.) To test: 0. Apply patch 1. koha-indexer --stop kohadev 2. vi /etc/default/koha-common 3. Update ALTERNATE_INDEXER_DAEMON to be "/kohadevbox/koha/misc/zebra_indexer.pl" 4. koha-indexer --start kohadev 5. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 6. Change 245$a to "Gone" instead of "Gairm" and save the record 7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Gairm 8. Repeat Step 7 for 5+ seconds until your search returns no results 9. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 10. Change 245$a from "Gone" to "Gairm" 11. Repeat Step 7 for 5+ seconds until your search redirects you to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29&found1=1 To confirm lower memory usage: 12. ps -efww | grep "zebra_indexer" 13. top -p <PID of zebra_indexer.pl process> 14. The top tool should show about 6.1MB usage. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joonas.kylmala@helsinki.fi --- Comment #11 from Joonas Kylmälä <joonas.kylmala@helsinki.fi> --- I would be bit hesitant to take this into koha because it seems like this will break very easily due to not using any of the abstractions we have. Also, there is no license header. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |julian.maurice@biblibre.com --- Comment #13 from Julian Maurice <julian.maurice@biblibre.com> ---

14. The top tool should show about 6.1MB usage.

I get 6.9MB, and there is an additional 17.6MB used by the child process every 5 seconds. So that's 24.5MB taken only to check if there is records to index. Still better than 180MB, but worth noting IMO. Talking about that, I wonder if it's really better to use a child process in this case. Instead of just staying in memory, the code will be loaded/unloaded every 5 seconds. Maybe it will use less memory 99% of the time, but it will probably use more cpu time (and it will need all the memory once every 5 second anyway, so that memory is not really "free") I tried different things to measure memory consumption depending on what we use: - if I add 'require C4::Context' to _get_count (and nothing else), the child process memory consumption goes up to 65MB - if I add 'C4::Context->dbh()', memory consumption goes up to 98MB I don't know exactly where the problem is (though I suspect the DBIx::Class schema to take a significant part of memory), but it would be useful for such cases to have a method that returns a DBI object without loading half of Koha. Also I agree with Joonas, the "$dbh creation" code should not be duplicated. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to Julian Maurice from comment #13)

...

14. The top tool should show about 6.1MB usage.

I get 6.9MB, and there is an additional 17.6MB used by the child process every 5 seconds. So that's 24.5MB taken only to check if there is records to index. Still better than 180MB, but worth noting IMO. Talking about that, I wonder if it's really better to use a child process in this case. Instead of just staying in memory, the code will be loaded/unloaded every 5 seconds. Maybe it will use less memory 99% of the time, but it will probably use more cpu time (and it will need all the memory once every 5 second anyway, so that memory is not really "free")

We could add an option. Personally, I rather not have the memory reserved by a process which is doing a little check. I much rather have it compete and allow more important processes to reserve memory. We could add an option to do the do the indexing check in the same process rather than using a child process, if you prefer.

I tried different things to measure memory consumption depending on what we use: - if I add 'require C4::Context' to _get_count (and nothing else), the child process memory consumption goes up to 65MB - if I add 'C4::Context->dbh()', memory consumption goes up to 98MB

I don't know exactly where the problem is (though I suspect the DBIx::Class schema to take a significant part of memory), but it would be useful for such cases to have a method that returns a DBI object without loading half of Koha.

I 100% agree that it would be useful to have a method that returns a DBI object without loading half of Koha.

Also I agree with Joonas, the "$dbh creation" code should not be duplicated.

In theory, yes. In practice, we're not at that point. I'm happy for this to stay as Failed QA, until someone else provides that minimal database connection object. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- (In reply to Joonas Kylmälä from comment #12)

(In reply to Joonas Kylmälä from comment #11)

...

I would be bit hesitant to take this into koha because it seems like this will break very easily due to not using any of the abstractions we have. Also, there is no license header.

After reading through Julian's comments, I think that I understand yours better. The abstractions are too resource intensive to be practical in this case. But I will admit that my code is not ideal. It's practical but does add some potential technical debt. A compromise would be to add core Koha methods that have minimal dependencies but are easier to maintain. I think that I'll just maintain this custom indexer locally though. I don't have time to add those core Koha methods at this point. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- (In reply to Julian Maurice from comment #18)

David, can you take a look at bug 28306 ? I think you can replace your _get_dbh call by a call to Koha::Database->dbh without using much more memory.

Thanks for commenting here. I saw bug 28306 and thought it would be perfect to use here. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |28410 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28410 [Bug 28410] [Omnibus] Reduce memory footprint -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 --- Comment #21 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #20)

I need to revisit this one at some point.

I had forgotten to enable it on one of my servers, and I just saved so much reserved memory as a result.

We mostly use Elasticsearch now, so I'm not sure if I'll revisit this one... -- You are receiving this mail because: You are watching all bug changes.