Memory footprint of Koha processes
Howdy! I'm trying to run a server with MANY koha instances (for educational purposes), but although the usage is very small, the resources required just make everything explode. background_jobs_worker.pl for instance uses 170M per instance, and plack uses 200M without anything being loaded. Any suggestion how this could be hacked (locally) to allow for 200 koha instances on a machine without requiring 100G of memory ? The object model seems at fault here, but I see no way to go around "use Koha::BackgroundJobs;" at the top of the script. Thanks! -- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
PS this is 22.05.06 Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2023-10-23 15:57, Philippe Blouin wrote:
Howdy!
I'm trying to run a server with MANY koha instances (for educational purposes), but although the usage is very small, the resources required just make everything explode.
background_jobs_worker.pl for instance uses 170M per instance, and plack uses 200M without anything being loaded.
Any suggestion how this could be hacked (locally) to allow for 200 koha instances on a machine without requiring 100G of memory ? The object model seems at fault here, but I see no way to go around "use Koha::BackgroundJobs;" at the top of the script.
Thanks!
-- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
Hi Philippe, I run quite a few Koha instances, so I feel your pain. A couple months ago I opened https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34477 but I haven’t dedicated any time to work on it yet (and I have no plans to work on it anytime soon). The ORM eats up a lot of memory, and if I recall correctly C4::Context also requires a fair bit of memory. I don’t think you’ll really be able to hack this one per se as there are quite a few changes that are needed. That said… I suppose you could always create an alternative background jobs worker and use that instead. If I recall correctly, I don’t think you use the RabbitMQ, so you could exclude lots of code. In theory Koha::Database->dbh() should provide a database handle with a smaller memory footprint, but I haven’t tested it. But yeah… I think you’d basically need to do a rewrite. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Philippe Blouin via Koha-devel Sent: Tuesday, 24 October 2023 6:57 AM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] Memory footprint of Koha processes Howdy! I'm trying to run a server with MANY koha instances (for educational purposes), but although the usage is very small, the resources required just make everything explode. background_jobs_worker.pl for instance uses 170M per instance, and plack uses 200M without anything being loaded. Any suggestion how this could be hacked (locally) to allow for 200 koha instances on a machine without requiring 100G of memory ? The object model seems at fault here, but I see no way to go around <useKoha::BackgroundJobs;> "use Koha::BackgroundJobs;" at the top of the script. Thanks! -- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 <mailto:philippe.blouin@inLibro.com> philippe.blouin@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com
Some ideas: 1. You could hack the worker to process all queues, then you could have only one worker (?) 2. Stop the worker, modify Koha::BackgroundJob->enqueue to call ->process (not with $self, you will need to refetch the job) 3. You can remove the "use" statements from plack.psgi to reduce plack's memory footprint (but will be slower then) 4. Investigate why the worker is now taking 150M while it was 100 after 28413 5. Care about my bugs when I am working on memory footprint, so that I don't have the feeling to do something nobody cares about... 28410 ;) Le lun. 23 oct. 2023 à 21:57, Philippe Blouin via Koha-devel < koha-devel@lists.koha-community.org> a écrit :
Howdy!
I'm trying to run a server with MANY koha instances (for educational purposes), but although the usage is very small, the resources required just make everything explode.
background_jobs_worker.pl for instance uses 170M per instance, and plack uses 200M without anything being loaded.
Any suggestion how this could be hacked (locally) to allow for 200 koha instances on a machine without requiring 100G of memory ? The object model seems at fault here, but I see no way to go around "use Koha::BackgroundJobs;" at the top of the script.
Thanks! -- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Hi! On Mon, Oct 23, 2023 at 03:57:16PM -0400, Philippe Blouin via Koha-devel wrote:
background_jobs_worker.pl for instance uses 170M per instance, and plack uses 200M without anything being loaded.
How are you calculating the memory usage? I remember from mod_perl times that if you preload as much code as possible before forking of plack/redis workers much of the memory will be shared between processes. So maybe the actual memory usage is much lower than what you expect it to be. And if it's not, then we should investigate how to pull as much `use` statements into pieces of code that are loaded before a `fork` happens. Greetings, domm -- Thomas Klausner domm https://domm.plix.at Just another ( Perl | Postgres | Bicycle | Food | Photo | Vinyl ) Hacker Moneglia, Italy, via Puck, 2 day exposure [ 2023-10-14 10:00 > https://domm.plix.at/potd/2023-10-14.html ] Long and good read: http://bit-player.org/2023/ai-and-the-end-of-programming [ 2023-09-14 12:01 > https://domm.plix.at/microblog.html ]
Thanks everyone for the replies! I will give a look at everything, because I'm in dire need of solutions. Just a cue on the tests I did yesterday: #!/usr/bin/perl use Modern::Perl; use C4::Context; use Koha::BackgroundJobs; while (1){ sleep 10; } I use htop to see the memory usage. The RES is the value I care about. And the total memory usage of the machine tells me it is indeed additive (unlike mariadb or apache that show me the same value for each thread) Now, the code above uses 50K on _some_ instances, and 150M on others. I still haven't figured out the why for the discrepency, and I've tried about 20 instances, our our debian servers, our clients, and my coworkers ubuntu's. Something in the config? The 50K happens more often in 23.05, but when I restart them, they go to 150M and stay there. I'll be examining the koha-conf.xml today... As for plack, I have a very bare understanding on how it works. I suppose it loads all the code, so I suppose the 200M is justified? Of course, making it slower makes it useless, so I won't go that route. Thanks again, all! I'll be checking the BZ you suggested. Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2023-10-24 03:25, Thomas Klausner via Koha-devel wrote:
Hi!
On Mon, Oct 23, 2023 at 03:57:16PM -0400, Philippe Blouin via Koha-devel wrote:
background_jobs_worker.pl for instance uses 170M per instance, and plack uses 200M without anything being loaded. How are you calculating the memory usage? I remember from mod_perl times that if you preload as much code as possible before forking of plack/redis workers much of the memory will be shared between processes.
So maybe the actual memory usage is much lower than what you expect it to be. And if it's not, then we should investigate how to pull as much `use` statements into pieces of code that are loaded before a `fork` happens.
Greetings, domm
participants (4)
-
David Cook -
Jonathan Druart -
Philippe Blouin -
Thomas Klausner