https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36546 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- While this doesn't have to do with preloading per se, I am seeing a slowdown: I'm noticing on startup that each Starman worker spending some time doing some web requests... which are probably related to JSON::Validator. IP addresses: - 104.26.3.209 - 104.26.2.209 - 172.67.71.88 These are the IPv4 addresses for json-schema.org. I notice the first web request gets redirected as well: write(21, "GET /draft-04/schema HTTP/1.1\r\nUser-Agent: Mojolicious (Perl)\r\nAccept-Encoding: gzip\r\nHost: json-schema.org\r\n\r\n", 111) = 111 read(21, "HTTP/1.1 301 Moved Permanently\r\nDate: Tue, 30 Apr 2024 02:27:33 GMT\r\nContent-Type: text/html\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nCache-Control: max-age=3600\r\nExpires: Tue, 30 Apr 2024 03:27:33 GMT\r \nLocation: https://json-schema.org/draft-04/schema\r\nReport-To: {\"endpoints\":[{\"url\":\"h"..., 131072) = 870 -- Back in July 2020, there was a problem with swagger.io, which caused Koha to fail to startup. So in bug 23653 we started using a local copy of the schema. But in Bug 30194 it looks like that undid bug 23653... but that was for swagger.io. Note if json-schema.org is down (just map json-schema.org to localhost to reproduce), Koha appears to start, but the API doesn't work. You'll get 404 errors for every API endpoint. -- It looks like we can provide our own string to the $schema->specification(), so we can restore the idea behind bug 23653. That should insulate us from issues with json-schema.org and make for faster startup times. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Created attachment 165826 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=165826&action=edit Bug 36721: Add -M Koha::Preload to preload libraries into starman master By preloading common libraries into starman master, we save memory as starman master will share the memory addresses with its starman worker processes. We will also save time because each individual worker won't need to load these common modules. They'll already be done! To test: 0. Don't apply the patch yet 1. In koha-conf.xml, set the plack workers to a high number 2. Restart Plack 3. Note the memory usage using "free -m" 4. Apply the patch 5. Restart Plack using "debian/scripts/koha-plack" 6. Note the memory usage using "free -m" 7. Memory usage should be significantly less (around 1GB with 16 workers) 8. Comparing time is more difficult, but can be done before/after patch using strace, top, or by modifying Koha/REST/V1.pm to output a timestamp at the end of its load. Other ideas are also welcome. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (Note: I preload an entire Catalyst app in mod_perl, and it makes for lightning fast worker startup, as all the slow stuff is done in the master process before forking.) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Note that in https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/497#n... Victor and I talk a bit about using --preload-app with Starman to preload the entire app. The warnings against doing this on Starman documentation make me reluctant to do that, but we could investigate that too. Plackup seems to preload the whole app, so if we use that in dev and don't have any issues... it would be tempting to start preloading the app in prod as well, because it would likely have good performance benefits. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |domm@plix.at -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- I had another crack at this... I watched a "starman worker" startup using a hardcoded delay and strace, and I took note of every non-Koha Perl module that gets loaded in startup, and there are about 90. __Memory savings__ After adding the 90 modules to Koha::Preload, I did manage to get a significant drop in RAM used across a large number of instances (about 4-5GB), even when each instance has a small number of workers. __Startup time__ The more interesting thing to note was that "koha-plack --restart --quiet $(koha-list --enabled --plack))" took 4 minutes instead of 1 minute. So preloading a lot of modules actually made the overall restart slower! I think that's because koha-plack only moves on to the next instance once the starman master has launched, and it will be slower to launch since it's loading a lot of modules. __Startup load__ While the instances with preloaded modules were slower to load, the load was much much lower. A load of 2 where 2 workers are used. And that was very stable. __Reload time/load__ Without preloading, you can reload a large number of instances in about 30 seconds, but it puts a huge load on the system. With preloading, it seems the same amount of time and same load to reload. At least in terms of time reported by the "koha-plack" tool. Anecdotally, it seems like the actual reload time is faster overall, which would make sense. -- So overall... there's pros and cons. Preloading modules means lower memory use and higher stability, but it does mean slower starts/restarts. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36721 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=28411 Blocks| |28410 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28410 [Bug 28410] [Omnibus] Reduce memory footprint -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.