Optimizing Starman startup
Hi all, Do you despair when you see the following periodically in "top" when a starman worker is recreated ? PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl Or the following in top when you install koha-common package or restart the koha-common service? 11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl 11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl 11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl 11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl 11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl 11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl 11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl 11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker): my $validator = JSON::Validator::OpenAPI::Mojolicious->new; $validator->load_and_validate_schema( $self->home->rel_file("api/v1/swagger/swagger.json"), { allow_invalid_ref => 1, } ); push @{$self->routes->namespaces}, 'Koha::Plugin'; my $spec = $validator->schema->data; $self->plugin( 'Koha::REST::Plugin::PluginRoutes' => { spec => $spec, validator => $validator } ); $self->plugin( OpenAPI => { spec => $spec, route => $self->routes->under('/api/v1')->to('Auth#under'), allow_invalid_ref => 1, # required by our spec because $ref directly under # Paths-, Parameters-, Definitions- & Info-object # is not allowed by the OpenAPI specification. } ); Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it's done 1 time per Starman master instance rather than 1 time per Starman worker instance? I find "/usr/share/koha/api/v1/app.pl" to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595
Hi, I did some work on improving it here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to 1.) avoid doing it too often 2.) cache the results and avoid doing it if results are cached If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. --Ere dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
Hi Ere, Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.) It does seem like changes to the OpenAPI plugin would be needed for caching. I'm going to try backporting your change and try pre-loading and see how far that gets me. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi, I did some work on improving it here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to 1.) avoid doing it too often 2.) cache the results and avoid doing it if results are cached If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. --Ere dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Thanks for that optimization. My restart time has gone from 2 minutes 45 seconds to 1 minute 30 seconds. Much better. Now I'm going to try the preload and see if I can get it down to 45 seconds or better... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:28 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi Ere, Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.) It does seem like changes to the OpenAPI plugin would be needed for caching. I'm going to try backporting your change and try pre-loading and see how far that gets me. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi, I did some work on improving it here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to 1.) avoid doing it too often 2.) cache the results and avoid doing it if results are cached If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. --Ere dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
That was an interesting experience. When you run koha-plack --start, it spins through a long list of instances quickly, because it returns immediately after starting the Starman master, which doesn't really do much, but fork off worker processes, which then consume a huge amount of resources concurrently. However, when you preload the app using --preload-app, the resource consumption is in the Starman master, and the command doesn't return until after the Starman master has loaded the app. So you synchronously move through the list 1 by 1. This has pros and cons. Pro: You can start 60 Koha instances with very little resource usage, because you're only starting 1 Koha at a time. Yay! Con: "service koha-common restart" will stop all Plack instances and then start them all. This is suboptimal, as it took me about 3.5 minutes to start 60 Koha instances. That's a lot of downtime for the last Koha instance in that list. Of course, we could change "koha-common" init service to actually restart instead of stop/start on a "restart" action. But the other gotcha is Debian package installs/upgrades (hello continuous deployment). According to https://manpages.debian.org/testing/debhelper/dh_installinit.1.en.html, the init script is "stopped" in the prerm and "started" in the postinst. The documentation is not clear if it's at the start or end of postinst...) Based on my review of syslog and /var/log/apt/history.log, it looks like it's actually started at the beginning of postinst, which is before koha-schema-upgrade actually runs. I suppose that makes sense as that should put Koha into maintenance mode until the schema upgrade completes... Operations is so not fun heh. Really at the end of the day I guess it comes back to trying to optimize that OpenAPI plugin to reduce the resource consumption and resulting downtime. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:46 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Thanks for that optimization. My restart time has gone from 2 minutes 45 seconds to 1 minute 30 seconds. Much better. Now I'm going to try the preload and see if I can get it down to 45 seconds or better... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:28 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi Ere, Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.) It does seem like changes to the OpenAPI plugin would be needed for caching. I'm going to try backporting your change and try pre-loading and see how far that gets me. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi, I did some work on improving it here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to 1.) avoid doing it too often 2.) cache the results and avoid doing it if results are cached If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. --Ere dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Looking at newer versions of Mojolicious::Plugin::OpenAPI and the coding choices are... interesting. I opened a ticket against JSON::Validator (https://github.com/jhthorsen/json-validator/issues/246), but maybe I should open it against Mojolicious::Plugin::OpenAPI. We'll see how responsive the author is on Github. Of course, we're using such ancient versions of the Mojolicious modules that even convincing the author to cache validation or skip validation won't really help us in the short- to medium- term. At the moment, I think that I might just have to resign myself to Koha being slow to start up, as we're beholden to this third-party module. That said, we could fork Mojolicious::Plugin::OpenAPI. I suppose I could just do that locally too. Hmmmm. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 12:12 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup That was an interesting experience. When you run koha-plack --start, it spins through a long list of instances quickly, because it returns immediately after starting the Starman master, which doesn't really do much, but fork off worker processes, which then consume a huge amount of resources concurrently. However, when you preload the app using --preload-app, the resource consumption is in the Starman master, and the command doesn't return until after the Starman master has loaded the app. So you synchronously move through the list 1 by 1. This has pros and cons. Pro: You can start 60 Koha instances with very little resource usage, because you're only starting 1 Koha at a time. Yay! Con: "service koha-common restart" will stop all Plack instances and then start them all. This is suboptimal, as it took me about 3.5 minutes to start 60 Koha instances. That's a lot of downtime for the last Koha instance in that list. Of course, we could change "koha-common" init service to actually restart instead of stop/start on a "restart" action. But the other gotcha is Debian package installs/upgrades (hello continuous deployment). According to https://manpages.debian.org/testing/debhelper/dh_installinit.1.en.html, the init script is "stopped" in the prerm and "started" in the postinst. The documentation is not clear if it's at the start or end of postinst...) Based on my review of syslog and /var/log/apt/history.log, it looks like it's actually started at the beginning of postinst, which is before koha-schema-upgrade actually runs. I suppose that makes sense as that should put Koha into maintenance mode until the schema upgrade completes... Operations is so not fun heh. Really at the end of the day I guess it comes back to trying to optimize that OpenAPI plugin to reduce the resource consumption and resulting downtime. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:46 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Thanks for that optimization. My restart time has gone from 2 minutes 45 seconds to 1 minute 30 seconds. Much better. Now I'm going to try the preload and see if I can get it down to 45 seconds or better... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:28 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi Ere, Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.) It does seem like changes to the OpenAPI plugin would be needed for caching. I'm going to try backporting your change and try pre-loading and see how far that gets me. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi, I did some work on improving it here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to 1.) avoid doing it too often 2.) cache the results and avoid doing it if results are cached If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. --Ere dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
It looks like newer version of the plugin might have cached spec/schema functionality, but I'm not finding the plugin author very helpful so far. My guess is that maybe... maybe... Mojolicious::Plugin::OpenAPI 3.41+ with JSON::Validator 4.10+ might perform better. But it's just a guess at this point. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 1:21 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Looking at newer versions of Mojolicious::Plugin::OpenAPI and the coding choices are... interesting. I opened a ticket against JSON::Validator (https://github.com/jhthorsen/json-validator/issues/246), but maybe I should open it against Mojolicious::Plugin::OpenAPI. We'll see how responsive the author is on Github. Of course, we're using such ancient versions of the Mojolicious modules that even convincing the author to cache validation or skip validation won't really help us in the short- to medium- term. At the moment, I think that I might just have to resign myself to Koha being slow to start up, as we're beholden to this third-party module. That said, we could fork Mojolicious::Plugin::OpenAPI. I suppose I could just do that locally too. Hmmmm. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 12:12 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup That was an interesting experience. When you run koha-plack --start, it spins through a long list of instances quickly, because it returns immediately after starting the Starman master, which doesn't really do much, but fork off worker processes, which then consume a huge amount of resources concurrently. However, when you preload the app using --preload-app, the resource consumption is in the Starman master, and the command doesn't return until after the Starman master has loaded the app. So you synchronously move through the list 1 by 1. This has pros and cons. Pro: You can start 60 Koha instances with very little resource usage, because you're only starting 1 Koha at a time. Yay! Con: "service koha-common restart" will stop all Plack instances and then start them all. This is suboptimal, as it took me about 3.5 minutes to start 60 Koha instances. That's a lot of downtime for the last Koha instance in that list. Of course, we could change "koha-common" init service to actually restart instead of stop/start on a "restart" action. But the other gotcha is Debian package installs/upgrades (hello continuous deployment). According to https://manpages.debian.org/testing/debhelper/dh_installinit.1.en.html, the init script is "stopped" in the prerm and "started" in the postinst. The documentation is not clear if it's at the start or end of postinst...) Based on my review of syslog and /var/log/apt/history.log, it looks like it's actually started at the beginning of postinst, which is before koha-schema-upgrade actually runs. I suppose that makes sense as that should put Koha into maintenance mode until the schema upgrade completes... Operations is so not fun heh. Really at the end of the day I guess it comes back to trying to optimize that OpenAPI plugin to reduce the resource consumption and resulting downtime. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:46 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Thanks for that optimization. My restart time has gone from 2 minutes 45 seconds to 1 minute 30 seconds. Much better. Now I'm going to try the preload and see if I can get it down to 45 seconds or better... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:28 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi Ere, Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.) It does seem like changes to the OpenAPI plugin would be needed for caching. I'm going to try backporting your change and try pre-loading and see how far that gets me. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi, I did some work on improving it here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to 1.) avoid doing it too often 2.) cache the results and avoid doing it if results are cached If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. --Ere dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
I've always found the OpenAPI Dev finally and responsive. However, it's worth remembering were really not using the module as it's intended. Mojolicious D the ecosystem of modules around it are generally fast moving and we don't keep up at all.. we also deliver into the guts frequently and do things the authors never expected and finally we not using any of the recommended deployment options. Hypnotoad is by far the most optimal way of running mojo and this module. We have so many layers of obfuscation in our code it's hard to comprehend.. various layers virtualizing other layers etc. On Wed, 21 Apr 2021, 6:23 am , <dcook@prosentient.com.au> wrote:
It looks like newer version of the plugin might have cached spec/schema functionality, but I'm not finding the plugin author very helpful so far. My guess is that maybe... maybe... Mojolicious::Plugin::OpenAPI 3.41+ with JSON::Validator 4.10+ might perform better. But it's just a guess at this point.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 1:21 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Looking at newer versions of Mojolicious::Plugin::OpenAPI and the coding choices are... interesting.
I opened a ticket against JSON::Validator ( https://github.com/jhthorsen/json-validator/issues/246), but maybe I should open it against Mojolicious::Plugin::OpenAPI. We'll see how responsive the author is on Github.
Of course, we're using such ancient versions of the Mojolicious modules that even convincing the author to cache validation or skip validation won't really help us in the short- to medium- term.
At the moment, I think that I might just have to resign myself to Koha being slow to start up, as we're beholden to this third-party module.
That said, we could fork Mojolicious::Plugin::OpenAPI.
I suppose I could just do that locally too. Hmmmm.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 12:12 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
That was an interesting experience.
When you run koha-plack --start, it spins through a long list of instances quickly, because it returns immediately after starting the Starman master, which doesn't really do much, but fork off worker processes, which then consume a huge amount of resources concurrently.
However, when you preload the app using --preload-app, the resource consumption is in the Starman master, and the command doesn't return until after the Starman master has loaded the app. So you synchronously move through the list 1 by 1.
This has pros and cons.
Pro: You can start 60 Koha instances with very little resource usage, because you're only starting 1 Koha at a time. Yay! Con: "service koha-common restart" will stop all Plack instances and then start them all. This is suboptimal, as it took me about 3.5 minutes to start 60 Koha instances. That's a lot of downtime for the last Koha instance in that list.
Of course, we could change "koha-common" init service to actually restart instead of stop/start on a "restart" action.
But the other gotcha is Debian package installs/upgrades (hello continuous deployment). According to https://manpages.debian.org/testing/debhelper/dh_installinit.1.en.html, the init script is "stopped" in the prerm and "started" in the postinst. The documentation is not clear if it's at the start or end of postinst...) Based on my review of syslog and /var/log/apt/history.log, it looks like it's actually started at the beginning of postinst, which is before koha-schema-upgrade actually runs. I suppose that makes sense as that should put Koha into maintenance mode until the schema upgrade completes...
Operations is so not fun heh.
Really at the end of the day I guess it comes back to trying to optimize that OpenAPI plugin to reduce the resource consumption and resulting downtime.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:46 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Thanks for that optimization. My restart time has gone from 2 minutes 45 seconds to 1 minute 30 seconds. Much better.
Now I'm going to try the preload and see if I can get it down to 45 seconds or better...
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:28 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
He did answer back fairly quickly but not a very helpful way. Maybe I just wasn’t asking the right questions. Agreed. I wish that we kept up better with Mojolicious. I was actually wondering why we don’t use Hypnotoad for the REST API. Is it just because of the extra process management? I recall someone (Julian or Kyle?) did some work to make Koha into a Mojolicious application and I have some pending work that uses Mojolicious more, so it would be nice to further embrace it and to use Hypnotoad. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Renvoize, Martin <martin.renvoize@ptfs-europe.com> Sent: Wednesday, 21 April 2021 4:40 PM To: David Cook <dcook@prosentient.com.au> Cc: Koha Devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup I've always found the OpenAPI Dev finally and responsive. However, it's worth remembering were really not using the module as it's intended. Mojolicious D the ecosystem of modules around it are generally fast moving and we don't keep up at all.. we also deliver into the guts frequently and do things the authors never expected and finally we not using any of the recommended deployment options. Hypnotoad is by far the most optimal way of running mojo and this module. We have so many layers of obfuscation in our code it's hard to comprehend.. various layers virtualizing other layers etc. On Wed, 21 Apr 2021, 6:23 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: It looks like newer version of the plugin might have cached spec/schema functionality, but I'm not finding the plugin author very helpful so far. My guess is that maybe... maybe... Mojolicious::Plugin::OpenAPI 3.41+ with JSON::Validator 4.10+ might perform better. But it's just a guess at this point. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> Sent: Wednesday, 21 April 2021 1:21 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> >; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Looking at newer versions of Mojolicious::Plugin::OpenAPI and the coding choices are... interesting. I opened a ticket against JSON::Validator (https://github.com/jhthorsen/json-validator/issues/246), but maybe I should open it against Mojolicious::Plugin::OpenAPI. We'll see how responsive the author is on Github. Of course, we're using such ancient versions of the Mojolicious modules that even convincing the author to cache validation or skip validation won't really help us in the short- to medium- term. At the moment, I think that I might just have to resign myself to Koha being slow to start up, as we're beholden to this third-party module. That said, we could fork Mojolicious::Plugin::OpenAPI. I suppose I could just do that locally too. Hmmmm. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> Sent: Wednesday, 21 April 2021 12:12 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> >; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup That was an interesting experience. When you run koha-plack --start, it spins through a long list of instances quickly, because it returns immediately after starting the Starman master, which doesn't really do much, but fork off worker processes, which then consume a huge amount of resources concurrently. However, when you preload the app using --preload-app, the resource consumption is in the Starman master, and the command doesn't return until after the Starman master has loaded the app. So you synchronously move through the list 1 by 1. This has pros and cons. Pro: You can start 60 Koha instances with very little resource usage, because you're only starting 1 Koha at a time. Yay! Con: "service koha-common restart" will stop all Plack instances and then start them all. This is suboptimal, as it took me about 3.5 minutes to start 60 Koha instances. That's a lot of downtime for the last Koha instance in that list. Of course, we could change "koha-common" init service to actually restart instead of stop/start on a "restart" action. But the other gotcha is Debian package installs/upgrades (hello continuous deployment). According to https://manpages.debian.org/testing/debhelper/dh_installinit.1.en.html, the init script is "stopped" in the prerm and "started" in the postinst. The documentation is not clear if it's at the start or end of postinst...) Based on my review of syslog and /var/log/apt/history.log, it looks like it's actually started at the beginning of postinst, which is before koha-schema-upgrade actually runs. I suppose that makes sense as that should put Koha into maintenance mode until the schema upgrade completes... Operations is so not fun heh. Really at the end of the day I guess it comes back to trying to optimize that OpenAPI plugin to reduce the resource consumption and resulting downtime. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> Sent: Wednesday, 21 April 2021 10:46 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> >; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Thanks for that optimization. My restart time has gone from 2 minutes 45 seconds to 1 minute 30 seconds. Much better. Now I'm going to try the preload and see if I can get it down to 45 seconds or better... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> Sent: Wednesday, 21 April 2021 10:28 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> >; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Hi Ere, Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.) It does seem like changes to the OpenAPI plugin would be needed for caching. I'm going to try backporting your change and try pre-loading and see how far that gets me. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Hi, I did some work on improving it here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to 1.) avoid doing it too often 2.) cache the results and avoid doing it if results are cached If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. --Ere dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl <http://app.pl>
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl <http://app.pl>
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl <http://app.pl>
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl <http://app.pl>
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl <http://app.pl>
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl <http://app.pl>
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl <http://app.pl>
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl <http://app.pl> ” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Le 21/04/2021 à 09:26, dcook@prosentient.com.au a écrit :
I recall someone (Julian or Kyle?) did some work to make Koha into a Mojolicious application
Yes, since 20.11 Koha can be started as a Mojolicious application. `hypnotoad bin/intranet` But you'll probably need https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26625 in order to configure the listen port/address. -- Julian Maurice BibLibre
It's too bad that we're not using Unix sockets out of the box (like we do with Starman) as that would make Bug 26625 less pressing but maybe I'll look more at that... The only downside with starting Koha as a Mojolicious application at the moment seems to be the experimentalness of it. I'm wondering how we can start a paradigm shift towards using it for production for everyone globally... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Julian Maurice Sent: Wednesday, 21 April 2021 7:00 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Le 21/04/2021 à 09:26, dcook@prosentient.com.au a écrit :
I recall someone (Julian or Kyle?) did some work to make Koha into a Mojolicious application
Yes, since 20.11 Koha can be started as a Mojolicious application. `hypnotoad bin/intranet` But you'll probably need https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26625 in order to configure the listen port/address. -- Julian Maurice BibLibre _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Awesome, good to hear it's been useful. :) Even better if you can improve it further. --Ere dcook@prosentient.com.au kirjoitti 21.4.2021 klo 3.46:
Thanks for that optimization. My restart time has gone from 2 minutes 45 seconds to 1 minute 30 seconds. Much better.
Now I'm going to try the preload and see if I can get it down to 45 seconds or better...
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Wednesday, 21 April 2021 10:28 AM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
Hi David, I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it. Thanks for taking a look! --Ere dcook@prosentient.com.au kirjoitti 21.4.2021 klo 3.28:
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
Hi Ere, I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins. Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about. At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS. I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind. Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi> Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup Hi David, I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it. Thanks for taking a look! --Ere dcook@prosentient.com.au kirjoitti 21.4.2021 klo 3.28:
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around. As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win. On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au> wrote:
Hi Ere,
I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins.
Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about.
At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS.
I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind.
Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment...
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi> Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi David,
I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it.
Thanks for taking a look!
--Ere
dcook@prosentient.com.au kirjoitti 21.4.2021 klo 3.28:
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
I think that I accidentally offended him, as he hasn’t responded to me since his initial response. I do wonder if reducing the number of references would help, although I wonder how easy that would be to do in practice. It looks like we have about 5767 lines of JSON all up as is… so it would probably get even bigger if we dereferenced them. Oh… here’s a thought. Why don’t we compile it? According to https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files..., you can maintain many different files, and then use something like swagger-cli to create a single built/compiled OpenAPI file. That way JSON::Validator wouldn’t need to resolve any references for the core API. I don’t know if the plugins have any $ref in them but I’m guessing not (just based on Coverflow). So that could be a big win. I’m working on other things at the moment, but I’m going to put that on my eternal list. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Renvoize, Martin <martin.renvoize@ptfs-europe.com> Sent: Friday, 23 April 2021 5:24 PM To: David Cook <dcook@prosentient.com.au> Cc: Koha Devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around. As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win. On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: Hi Ere, I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins. Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about. At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS. I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind. Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> > Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> ; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Hi David, I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it. Thanks for taking a look! --Ere dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 21.4.2021 klo 3.28:
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl <http://app.pl>
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl <http://app.pl>
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl <http://app.pl>
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl <http://app.pl>
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl <http://app.pl>
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl <http://app.pl>
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl <http://app.pl>
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl <http://app.pl> ” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
So I just tried the following… -- root@kohadevbox:koha(master)$ npm install -g swagger-cli /usr/bin/swagger-cli -> /usr/lib/node_modules/swagger-cli/swagger-cli.js npm WARN @apidevtools/swagger-parser@10.0.2 requires a peer of openapi-types@>=7 but none is installed. You must install peer dependencies yourself. + swagger-cli@4.0.4 added 46 packages from 27 contributors in 8.203s -- root@kohadevbox:koha(master)$ time swagger-cli bundle api/v1/swagger/swagger.json --outfile api/v1/swagger/openapi.json --type json Created api/v1/swagger/openapi.json from api/v1/swagger/swagger.json real 0m0.296s user 0m0.346s sys 0m0.032s openapi.json is 10891 lines long but it actually contains 741 $ref lines like "$ref": "#/definitions/error" and "$ref": "#/definitions/patron_extended_attribute". -- Now to do some benchmarking… I ran the following code: #!/usr/bin/perl use JSON::Validator::OpenAPI::Mojolicious; my $validator = JSON::Validator::OpenAPI::Mojolicious->new; my $spec = $validator->bundle({ replace => 1, schema => "api/v1/swagger/swagger.json", }); The first time I ran it… it took 1.343 seconds. The second time and subsequent times it took .354 seconds. (That’s using Ubuntu 20.04 and JSON Validator 3.14.) That suggests caching although I’m not sure where. I don’t see anything obvious in /usr/share/perl5/JSON/Validator/cache. Trying with openapi.json yields .280 seconds instead of .354 seconds. It’s faster, but not significantly. So that suggests that the problem is actually with Koha::REST::Plugin::PluginRoutes or Mojolicious::Plugin::OpenAPI more specifically… David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: dcook@prosentient.com.au <dcook@prosentient.com.au> Sent: Monday, 26 April 2021 11:24 AM To: 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> Cc: 'Koha Devel' <koha-devel@lists.koha-community.org> Subject: RE: [Koha-devel] Optimizing Starman startup I think that I accidentally offended him, as he hasn’t responded to me since his initial response. I do wonder if reducing the number of references would help, although I wonder how easy that would be to do in practice. It looks like we have about 5767 lines of JSON all up as is… so it would probably get even bigger if we dereferenced them. Oh… here’s a thought. Why don’t we compile it? According to https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files..., you can maintain many different files, and then use something like swagger-cli to create a single built/compiled OpenAPI file. That way JSON::Validator wouldn’t need to resolve any references for the core API. I don’t know if the plugins have any $ref in them but I’m guessing not (just based on Coverflow). So that could be a big win. I’m working on other things at the moment, but I’m going to put that on my eternal list. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Renvoize, Martin <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com> > Sent: Friday, 23 April 2021 5:24 PM To: David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > Cc: Koha Devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Optimizing Starman startup Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around. As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win. On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: Hi Ere, I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins. Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about. At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS. I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind. Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> > Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> ; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Hi David, I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it. Thanks for taking a look! --Ere dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 21.4.2021 klo 3.28:
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl <http://app.pl>
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl <http://app.pl>
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl <http://app.pl>
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl <http://app.pl>
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl <http://app.pl>
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl <http://app.pl>
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl <http://app.pl>
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl <http://app.pl> ” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
After some more experimenting, it’s clear that the problem isn’t JSON::Validator::OpenAPI::Mojolicious or Koha::REST::Plugin::PluginRoutes. If you exclude Mojolicious::Plugin::OpenAPI, the startup is very fast. It’s 30 seconds start to finish to restart 60 instances and each instance restarts very quickly. When using Mojolicious::Plugin::OpenAPI, it takes about 3 minutes and there’s a fair bit of downtime during that time. When I do a strace, I’m noticing that a process can spend 30 seconds just allocating memory for Mojolicious::Plugin::OpenAPI, but it only happens once you hit a certain volume of processes. If you’re just starting up 1 or 2, then it’s only a couple seconds. But if you have say 60-120 processes, it can take up to 30 seconds for Mojolicious::Plugin::OpenAPI to do its work. I’m putting 10 CPUs to this work, but clearly that’s not enough. I imagine there may be other bottlenecks accessing the memory as well. Has anyone profiled Mojolicious before? I’m guessing maybe Martin? I suspect that this is just a problem that I’m going to have to live with but maybe it is a case where I can find a way to optimize Mojolicious::Plugin::OpenAPI. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Monday, 26 April 2021 5:12 PM To: 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> Cc: 'Koha Devel' <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup So I just tried the following… -- root@kohadevbox:koha(master)$ npm install -g swagger-cli /usr/bin/swagger-cli -> /usr/lib/node_modules/swagger-cli/swagger-cli.js npm WARN @apidevtools/swagger-parser@10.0.2 requires a peer of openapi-types@>=7 but none is installed. You must install peer dependencies yourself. + swagger-cli@4.0.4 <mailto:swagger-cli@4.0.4> added 46 packages from 27 contributors in 8.203s -- root@kohadevbox:koha(master)$ time swagger-cli bundle api/v1/swagger/swagger.json --outfile api/v1/swagger/openapi.json --type json Created api/v1/swagger/openapi.json from api/v1/swagger/swagger.json real 0m0.296s user 0m0.346s sys 0m0.032s openapi.json is 10891 lines long but it actually contains 741 $ref lines like "$ref": "#/definitions/error" and "$ref": "#/definitions/patron_extended_attribute". -- Now to do some benchmarking… I ran the following code: #!/usr/bin/perl use JSON::Validator::OpenAPI::Mojolicious; my $validator = JSON::Validator::OpenAPI::Mojolicious->new; my $spec = $validator->bundle({ replace => 1, schema => "api/v1/swagger/swagger.json", }); The first time I ran it… it took 1.343 seconds. The second time and subsequent times it took .354 seconds. (That’s using Ubuntu 20.04 and JSON Validator 3.14.) That suggests caching although I’m not sure where. I don’t see anything obvious in /usr/share/perl5/JSON/Validator/cache. Trying with openapi.json yields .280 seconds instead of .354 seconds. It’s faster, but not significantly. So that suggests that the problem is actually with Koha::REST::Plugin::PluginRoutes or Mojolicious::Plugin::OpenAPI more specifically… David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > Sent: Monday, 26 April 2021 11:24 AM To: 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com> > Cc: 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: RE: [Koha-devel] Optimizing Starman startup I think that I accidentally offended him, as he hasn’t responded to me since his initial response. I do wonder if reducing the number of references would help, although I wonder how easy that would be to do in practice. It looks like we have about 5767 lines of JSON all up as is… so it would probably get even bigger if we dereferenced them. Oh… here’s a thought. Why don’t we compile it? According to https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files..., you can maintain many different files, and then use something like swagger-cli to create a single built/compiled OpenAPI file. That way JSON::Validator wouldn’t need to resolve any references for the core API. I don’t know if the plugins have any $ref in them but I’m guessing not (just based on Coverflow). So that could be a big win. I’m working on other things at the moment, but I’m going to put that on my eternal list. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Renvoize, Martin <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com> > Sent: Friday, 23 April 2021 5:24 PM To: David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > Cc: Koha Devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Optimizing Starman startup Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around. As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win. On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: Hi Ere, I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins. Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about. At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS. I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind. Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> > Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> ; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Hi David, I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it. Thanks for taking a look! --Ere dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 21.4.2021 klo 3.28:
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl <http://app.pl>
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl <http://app.pl>
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl <http://app.pl>
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl <http://app.pl>
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl <http://app.pl>
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl <http://app.pl>
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl <http://app.pl>
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl <http://app.pl> ” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
After more experimenting, I have concluded that JSON::Validator->validate() is the culprit in terms of CPU time and memory usage. Fortunately, I’ve determined that Mojo::JSON and Digest::MD5 can be used together to create consistent reproducible checksums, which could be used for caching validated schemas. Of course, a solution would involve changes to JSON::Validator (and possibly Mojolicious::Plugin::OpenAPI depending on the chosen solution), and then we’d have to wait for the new and improved version to come downstream, so we wouldn’t see the benefit of this for years. That said… we could always roll our own JSON::Validator. And if we don’t want to do it as a community, I could always just do it myself. In terms of testing… with 18 CPUs I can restart 60 instances (120 processes) and get through the app setup in about 60 seconds with significant server load, when using validation. Without validation, I can do it in 20 seconds without significant server load (beyond a few short-lived CPU spikes). I’m thinking about writing a patch and sending a pull request for JSON::Validator, but also really thinking about implementing it locally too at least in the meantime. I haven’t heard from the author of JSON::Validator for a little while now, but I hope I do hear back from him. I think it would be a great addition to the library. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of dcook@prosentient.com.au Sent: Monday, 26 April 2021 7:01 PM To: 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> Cc: 'Koha Devel' <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup After some more experimenting, it’s clear that the problem isn’t JSON::Validator::OpenAPI::Mojolicious or Koha::REST::Plugin::PluginRoutes. If you exclude Mojolicious::Plugin::OpenAPI, the startup is very fast. It’s 30 seconds start to finish to restart 60 instances and each instance restarts very quickly. When using Mojolicious::Plugin::OpenAPI, it takes about 3 minutes and there’s a fair bit of downtime during that time. When I do a strace, I’m noticing that a process can spend 30 seconds just allocating memory for Mojolicious::Plugin::OpenAPI, but it only happens once you hit a certain volume of processes. If you’re just starting up 1 or 2, then it’s only a couple seconds. But if you have say 60-120 processes, it can take up to 30 seconds for Mojolicious::Plugin::OpenAPI to do its work. I’m putting 10 CPUs to this work, but clearly that’s not enough. I imagine there may be other bottlenecks accessing the memory as well. Has anyone profiled Mojolicious before? I’m guessing maybe Martin? I suspect that this is just a problem that I’m going to have to live with but maybe it is a case where I can find a way to optimize Mojolicious::Plugin::OpenAPI. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> Sent: Monday, 26 April 2021 5:12 PM To: 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com> > Cc: 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Optimizing Starman startup So I just tried the following… -- root@kohadevbox:koha(master)$ npm install -g swagger-cli /usr/bin/swagger-cli -> /usr/lib/node_modules/swagger-cli/swagger-cli.js npm WARN @apidevtools/swagger-parser@10.0.2 requires a peer of openapi-types@>=7 but none is installed. You must install peer dependencies yourself. + swagger-cli@4.0.4 <mailto:swagger-cli@4.0.4> added 46 packages from 27 contributors in 8.203s -- root@kohadevbox:koha(master)$ time swagger-cli bundle api/v1/swagger/swagger.json --outfile api/v1/swagger/openapi.json --type json Created api/v1/swagger/openapi.json from api/v1/swagger/swagger.json real 0m0.296s user 0m0.346s sys 0m0.032s openapi.json is 10891 lines long but it actually contains 741 $ref lines like "$ref": "#/definitions/error" and "$ref": "#/definitions/patron_extended_attribute". -- Now to do some benchmarking… I ran the following code: #!/usr/bin/perl use JSON::Validator::OpenAPI::Mojolicious; my $validator = JSON::Validator::OpenAPI::Mojolicious->new; my $spec = $validator->bundle({ replace => 1, schema => "api/v1/swagger/swagger.json", }); The first time I ran it… it took 1.343 seconds. The second time and subsequent times it took .354 seconds. (That’s using Ubuntu 20.04 and JSON Validator 3.14.) That suggests caching although I’m not sure where. I don’t see anything obvious in /usr/share/perl5/JSON/Validator/cache. Trying with openapi.json yields .280 seconds instead of .354 seconds. It’s faster, but not significantly. So that suggests that the problem is actually with Koha::REST::Plugin::PluginRoutes or Mojolicious::Plugin::OpenAPI more specifically… David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > Sent: Monday, 26 April 2021 11:24 AM To: 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com> > Cc: 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: RE: [Koha-devel] Optimizing Starman startup I think that I accidentally offended him, as he hasn’t responded to me since his initial response. I do wonder if reducing the number of references would help, although I wonder how easy that would be to do in practice. It looks like we have about 5767 lines of JSON all up as is… so it would probably get even bigger if we dereferenced them. Oh… here’s a thought. Why don’t we compile it? According to https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files..., you can maintain many different files, and then use something like swagger-cli to create a single built/compiled OpenAPI file. That way JSON::Validator wouldn’t need to resolve any references for the core API. I don’t know if the plugins have any $ref in them but I’m guessing not (just based on Coverflow). So that could be a big win. I’m working on other things at the moment, but I’m going to put that on my eternal list. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Renvoize, Martin <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com> > Sent: Friday, 23 April 2021 5:24 PM To: David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > Cc: Koha Devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Optimizing Starman startup Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around. As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win. On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: Hi Ere, I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins. Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about. At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS. I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind. Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi> > Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> ; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Hi David, I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it. Thanks for taking a look! --Ere dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 21.4.2021 klo 3.28:
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master process pre-fork. There are warnings about doing that with open database connections, so we'd need to review plack.psgi, but a quick glance suggests it might be OK. (Alternatively, I have wondered about running the REST API as a separate process apart from Starman using hypnotoad. According to https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork preloads the application in the manager/master process, and Hypnotoad is based off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far that gets me.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of Ere Maijala Sent: Tuesday, 20 April 2021 4:48 PM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.
--Ere
dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 /usr/share/koha/api/v1/app.pl <http://app.pl>
Or the following in top when you install koha-common package or restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 /usr/share/koha/api/v1/app.pl <http://app.pl>
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 /usr/share/koha/api/v1/app.pl <http://app.pl>
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 /usr/share/koha/api/v1/app.pl <http://app.pl>
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 /usr/share/koha/api/v1/app.pl <http://app.pl>
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 /usr/share/koha/api/v1/app.pl <http://app.pl>
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 /usr/share/koha/api/v1/app.pl <http://app.pl>
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 /usr/share/koha/api/v1/app.pl <http://app.pl>
Well, I still have a lot of investigation left to do, but I notice 1 place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route => $self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the schema every time? Can we move the schema validation into a different module and preload it into Starman using the -M flag so that it’s done 1 time per Starman master instance rather than 1 time per Starman worker instance?
I find “/usr/share/koha/api/v1/app.pl <http://app.pl> ” to be the bane of deployments, as it puts a massive load on a server, when you have multiple Koha instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Ere Maijala Kansalliskirjasto / The National Library of Finland _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Thanks for digging up all the information. I think adding a caching validator would provide a nice improvement, so I'm in favor of it, and if I were to decide, I'd also include it in Koha. Still, getting it upstream would be great in the long run, so I'd support that as well. --Ere dcook@prosentient.com.au kirjoitti 27.4.2021 klo 5.34:
After more experimenting, I have concluded that JSON::Validator->validate() is the culprit in terms of CPU time and memory usage.
Fortunately, I’ve determined that Mojo::JSON and Digest::MD5 can be used together to create consistent reproducible checksums, which could be used for caching validated schemas.
Of course, a solution would involve changes to JSON::Validator (and possibly Mojolicious::Plugin::OpenAPI depending on the chosen solution), and then we’d have to wait for the new and improved version to come downstream, so we wouldn’t see the benefit of this for years.
That said… we could always roll our own JSON::Validator. And if we don’t want to do it as a community, I could always just do it myself.
In terms of testing… with 18 CPUs I can restart 60 instances (120 processes) and get through the app setup in about 60 seconds with significant server load, when using validation. Without validation, I can do it in 20 seconds without significant server load (beyond a few short-lived CPU spikes).
I’m thinking about writing a patch and sending a pull request for JSON::Validator, but also really thinking about implementing it locally too at least in the meantime.
I haven’t heard from the author of JSON::Validator for a little while now, but I hope I do hear back from him. I think it would be a great addition to the library.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *dcook@prosentient.com.au *Sent:* Monday, 26 April 2021 7:01 PM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org> *Subject:* Re: [Koha-devel] Optimizing Starman startup
After some more experimenting, it’s clear that the problem isn’t JSON::Validator::OpenAPI::Mojolicious or Koha::REST::Plugin::PluginRoutes. If you exclude Mojolicious::Plugin::OpenAPI, the startup is very fast. It’s 30 seconds start to finish to restart 60 instances and each instance restarts very quickly.
When using Mojolicious::Plugin::OpenAPI, it takes about 3 minutes and there’s a fair bit of downtime during that time.
When I do a strace, I’m noticing that a process can spend 30 seconds just allocating memory for Mojolicious::Plugin::OpenAPI, but it only happens once you hit a certain volume of processes. If you’re just starting up 1 or 2, then it’s only a couple seconds. But if you have say 60-120 processes, it can take up to 30 seconds for Mojolicious::Plugin::OpenAPI to do its work. I’m putting 10 CPUs to this work, but clearly that’s not enough. I imagine there may be other bottlenecks accessing the memory as well.
Has anyone profiled Mojolicious before? I’m guessing maybe Martin?
I suspect that this is just a problem that I’m going to have to live with but maybe it is a case where I can find a way to optimize Mojolicious::Plugin::OpenAPI.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org>> *On Behalf Of *dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> *Sent:* Monday, 26 April 2021 5:12 PM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* Re: [Koha-devel] Optimizing Starman startup
So I just tried the following…
--
root@kohadevbox:koha(master)$ npm install -g swagger-cli
/usr/bin/swagger-cli -> /usr/lib/node_modules/swagger-cli/swagger-cli.js
npm WARN @apidevtools/swagger-parser@10.0.2 requires a peer of openapi-types@>=7 but none is installed. You must install peer dependencies yourself.
+ swagger-cli@4.0.4 <mailto:swagger-cli@4.0.4>
added 46 packages from 27 contributors in 8.203s
--
root@kohadevbox:koha(master)$ time swagger-cli bundle api/v1/swagger/swagger.json --outfile api/v1/swagger/openapi.json --type json
Created api/v1/swagger/openapi.json from api/v1/swagger/swagger.json
real 0m0.296s
user 0m0.346s
sys 0m0.032s
openapi.json is 10891 lines long but it actually contains 741 $ref lines like "$ref": "#/definitions/error" and "$ref": "#/definitions/patron_extended_attribute".
--
Now to do some benchmarking… I ran the following code:
#!/usr/bin/perl
use JSON::Validator::OpenAPI::Mojolicious;
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
my $spec = $validator->bundle({
replace => 1,
schema => "api/v1/swagger/swagger.json",
});
The first time I ran it… it took 1.343 seconds. The second time and subsequent times it took .354 seconds. (That’s using Ubuntu 20.04 and JSON Validator 3.14.) That suggests caching although I’m not sure where. I don’t see anything obvious in /usr/share/perl5/JSON/Validator/cache.
Trying with openapi.json yields .280 seconds instead of .354 seconds. It’s faster, but not significantly.
So that suggests that the problem is actually with Koha::REST::Plugin::PluginRoutes or Mojolicious::Plugin::OpenAPI more specifically…
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> *Sent:* Monday, 26 April 2021 11:24 AM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* RE: [Koha-devel] Optimizing Starman startup
I think that I accidentally offended him, as he hasn’t responded to me since his initial response.
I do wonder if reducing the number of references would help, although I wonder how easy that would be to do in practice. It looks like we have about 5767 lines of JSON all up as is… so it would probably get even bigger if we dereferenced them.
Oh… here’s a thought. Why don’t we compile it? According to https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files... <https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files/>, you can maintain many different files, and then use something like swagger-cli to create a single built/compiled OpenAPI file.
That way JSON::Validator wouldn’t need to resolve any references for the core API. I don’t know if the plugins have any $ref in them but I’m guessing not (just based on Coverflow). So that could be a big win.
I’m working on other things at the moment, but I’m going to put that on my eternal list.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Renvoize, Martin <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Sent:* Friday, 23 April 2021 5:24 PM *To:* David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> *Cc:* Koha Devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* Re: [Koha-devel] Optimizing Starman startup
Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around.
As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win.
On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> wrote:
Hi Ere,
I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins.
Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about.
At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS.
I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind.
Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment...
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi>> Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup
Hi David,
I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it.
Thanks for taking a look!
--Ere
dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 21.4.2021 klo 3.28: > Hi Ere, > > Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. > > Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... <https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm> and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? > > Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. > > At a glance, we might be able to pre-load the app into the Starman > master process pre-fork. There are warnings about doing that with open > database connections, so we'd need to review plack.psgi, but a quick > glance suggests it might be OK. (Alternatively, I have wondered about > running the REST API as a separate process apart from Starman using > hypnotoad. According to > https://docs.mojolicious.org/Mojolicious/Guides/Cookbook <https://docs.mojolicious.org/Mojolicious/Guides/Cookbook>, > Mojo::Server::Prefork preloads the application in the manager/master > process, and Hypnotoad is based off that, so that would help.) > > It does seem like changes to the OpenAPI plugin would be needed for caching. > > I'm going to try backporting your change and try pre-loading and see how far that gets me. > > David Cook > Software Engineer > Prosentient Systems > Suite 7.03 > 6a Glen St > Milsons Point NSW 2061 > Australia > > Office: 02 9212 0899 > Online: 02 8005 0595 > > -----Original Message----- > From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org>> On > Behalf Of Ere Maijala > Sent: Tuesday, 20 April 2021 4:48 PM > To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Optimizing Starman startup > > Hi, > > I did some work on improving it here: > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700> > > That shaved a good bit of time from it, but it's still a heavy > operation, and it would make sense to > > 1.) avoid doing it too often > > 2.) cache the results and avoid doing it if results are cached > > If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. > > --Ere > > dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15: >> Hi all, >> >> Do you despair when you see the following periodically in “top” when >> a starman worker is recreated ? >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> >> 9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> Or the following in top when you install koha-common package or >> restart the koha-common service? >> >> 11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> Well, I still have a lot of investigation left to do, but I notice 1 >> place that a lot of time taken is here (per worker): >> >> my $validator = JSON::Validator::OpenAPI::Mojolicious->new; >> >> $validator->load_and_validate_schema( >> >> $self->home->rel_file("api/v1/swagger/swagger.json"), >> >> { >> >> allow_invalid_ref => 1, >> >> } >> >> ); >> >> push @{$self->routes->namespaces}, 'Koha::Plugin'; >> >> my $spec = $validator->schema->data; >> >> $self->plugin( >> >> 'Koha::REST::Plugin::PluginRoutes' => { >> >> spec => $spec, >> >> validator => $validator >> >> } >> >> ); >> >> $self->plugin( >> >> OpenAPI => { >> >> spec => $spec, >> >> route => >> $self->routes->under('/api/v1')->to('Auth#under'), >> >> allow_invalid_ref => >> >> 1, # required by our spec because $ref directly >> under >> >> # Paths-, Parameters-, Definitions- & >> Info-object >> >> # is not allowed by the OpenAPI specification. >> >> } >> >> ); >> >> Anyone have ideas for improving this? Do we have to validate the >> schema every time? Can we move the schema validation into a different >> module and preload it into Starman using the -M flag so that it’s >> done >> 1 time per Starman master instance rather than 1 time per Starman worker instance? >> >> I find “/usr/share/koha/api/v1/app.pl <http://app.pl>” to be the bane of deployments, >> as it puts a massive load on a server, when you have multiple Koha >> instances on the server. >> >> David Cook >> >> Software Engineer >> >> Prosentient Systems >> >> Suite 7.03 >> >> 6a Glen St >> >> Milsons Point NSW 2061 >> >> Australia >> >> Office: 02 9212 0899 >> >> Online: 02 8005 0595 >> >> >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> >> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : >> https://git.koha-community.org/ <https://git.koha-community.org/> bugs : >> https://bugs.koha-community.org/ <https://bugs.koha-community.org/> >> > > -- > Ere Maijala > Kansalliskirjasto / The National Library of Finland > _______________________________________________ > Koha-devel mailing list > Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> > website : https://www.koha-community.org/ <https://www.koha-community.org/> git : > https://git.koha-community.org/ <https://git.koha-community.org/> bugs : > https://bugs.koha-community.org/ <https://bugs.koha-community.org/> > >
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : https://git.koha-community.org/ <https://git.koha-community.org/> bugs : https://bugs.koha-community.org/ <https://bugs.koha-community.org/>
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
Based on my request, Jan has added an option to skip validation (https://github.com/jhthorsen/mojolicious-plugin-openapi/commit/673079d19f827...). However, he doesn't seem to want to implement my cache idea. He has said that he's willing to consider a PR, so I'll look at doing that some evening after work. He did propose freezing a JSON::Validator object and passing that between processes, but I don't really like that idea. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi> Sent: Tuesday, 27 April 2021 3:28 PM To: dcook@prosentient.com.au; 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> Cc: 'Koha Devel' <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Thanks for digging up all the information. I think adding a caching validator would provide a nice improvement, so I'm in favor of it, and if I were to decide, I'd also include it in Koha. Still, getting it upstream would be great in the long run, so I'd support that as well. --Ere dcook@prosentient.com.au kirjoitti 27.4.2021 klo 5.34:
After more experimenting, I have concluded that JSON::Validator->validate() is the culprit in terms of CPU time and memory usage.
Fortunately, I’ve determined that Mojo::JSON and Digest::MD5 can be used together to create consistent reproducible checksums, which could be used for caching validated schemas.
Of course, a solution would involve changes to JSON::Validator (and possibly Mojolicious::Plugin::OpenAPI depending on the chosen solution), and then we’d have to wait for the new and improved version to come downstream, so we wouldn’t see the benefit of this for years.
That said… we could always roll our own JSON::Validator. And if we don’t want to do it as a community, I could always just do it myself.
In terms of testing… with 18 CPUs I can restart 60 instances (120 processes) and get through the app setup in about 60 seconds with significant server load, when using validation. Without validation, I can do it in 20 seconds without significant server load (beyond a few short-lived CPU spikes).
I’m thinking about writing a patch and sending a pull request for JSON::Validator, but also really thinking about implementing it locally too at least in the meantime.
I haven’t heard from the author of JSON::Validator for a little while now, but I hope I do hear back from him. I think it would be a great addition to the library.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *dcook@prosentient.com.au *Sent:* Monday, 26 April 2021 7:01 PM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org> *Subject:* Re: [Koha-devel] Optimizing Starman startup
After some more experimenting, it’s clear that the problem isn’t JSON::Validator::OpenAPI::Mojolicious or Koha::REST::Plugin::PluginRoutes. If you exclude Mojolicious::Plugin::OpenAPI, the startup is very fast. It’s 30 seconds start to finish to restart 60 instances and each instance restarts very quickly.
When using Mojolicious::Plugin::OpenAPI, it takes about 3 minutes and there’s a fair bit of downtime during that time.
When I do a strace, I’m noticing that a process can spend 30 seconds just allocating memory for Mojolicious::Plugin::OpenAPI, but it only happens once you hit a certain volume of processes. If you’re just starting up 1 or 2, then it’s only a couple seconds. But if you have say 60-120 processes, it can take up to 30 seconds for Mojolicious::Plugin::OpenAPI to do its work. I’m putting 10 CPUs to this work, but clearly that’s not enough. I imagine there may be other bottlenecks accessing the memory as well.
Has anyone profiled Mojolicious before? I’m guessing maybe Martin?
I suspect that this is just a problem that I’m going to have to live with but maybe it is a case where I can find a way to optimize Mojolicious::Plugin::OpenAPI.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org>> *On Behalf Of *dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> *Sent:* Monday, 26 April 2021 5:12 PM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* Re: [Koha-devel] Optimizing Starman startup
So I just tried the following…
--
root@kohadevbox:koha(master)$ npm install -g swagger-cli
/usr/bin/swagger-cli -> /usr/lib/node_modules/swagger-cli/swagger-cli.js
npm WARN @apidevtools/swagger-parser@10.0.2 requires a peer of openapi-types@>=7 but none is installed. You must install peer dependencies yourself.
+ swagger-cli@4.0.4 <mailto:swagger-cli@4.0.4>
added 46 packages from 27 contributors in 8.203s
--
root@kohadevbox:koha(master)$ time swagger-cli bundle api/v1/swagger/swagger.json --outfile api/v1/swagger/openapi.json --type json
Created api/v1/swagger/openapi.json from api/v1/swagger/swagger.json
real 0m0.296s
user 0m0.346s
sys 0m0.032s
openapi.json is 10891 lines long but it actually contains 741 $ref lines like "$ref": "#/definitions/error" and "$ref": "#/definitions/patron_extended_attribute".
--
Now to do some benchmarking… I ran the following code:
#!/usr/bin/perl
use JSON::Validator::OpenAPI::Mojolicious;
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
my $spec = $validator->bundle({
replace => 1,
schema => "api/v1/swagger/swagger.json",
});
The first time I ran it… it took 1.343 seconds. The second time and subsequent times it took .354 seconds. (That’s using Ubuntu 20.04 and JSON Validator 3.14.) That suggests caching although I’m not sure where. I don’t see anything obvious in /usr/share/perl5/JSON/Validator/cache.
Trying with openapi.json yields .280 seconds instead of .354 seconds. It’s faster, but not significantly.
So that suggests that the problem is actually with Koha::REST::Plugin::PluginRoutes or Mojolicious::Plugin::OpenAPI more specifically…
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> *Sent:* Monday, 26 April 2021 11:24 AM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* RE: [Koha-devel] Optimizing Starman startup
I think that I accidentally offended him, as he hasn’t responded to me since his initial response.
I do wonder if reducing the number of references would help, although I wonder how easy that would be to do in practice. It looks like we have about 5767 lines of JSON all up as is… so it would probably get even bigger if we dereferenced them.
Oh… here’s a thought. Why don’t we compile it? According to https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple -files/ <https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multipl e-files/>, you can maintain many different files, and then use something like swagger-cli to create a single built/compiled OpenAPI file.
That way JSON::Validator wouldn’t need to resolve any references for the core API. I don’t know if the plugins have any $ref in them but I’m guessing not (just based on Coverflow). So that could be a big win.
I’m working on other things at the moment, but I’m going to put that on my eternal list.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Renvoize, Martin <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Sent:* Friday, 23 April 2021 5:24 PM *To:* David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> *Cc:* Koha Devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* Re: [Koha-devel] Optimizing Starman startup
Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around.
As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win.
On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> wrote:
Hi Ere,
I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins.
Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about.
At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS.
I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind.
Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment...
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi>> Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup
Hi David,
I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it.
Thanks for taking a look!
--Ere
dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 21.4.2021 klo 3.28: > Hi Ere, > > Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. > > Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... <https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm> and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? > > Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. > > At a glance, we might be able to pre-load the app into the Starman > master process pre-fork. There are warnings about doing that with open > database connections, so we'd need to review plack.psgi, but a quick > glance suggests it might be OK. (Alternatively, I have wondered about > running the REST API as a separate process apart from Starman using > hypnotoad. According to > https://docs.mojolicious.org/Mojolicious/Guides/Cookbook <https://docs.mojolicious.org/Mojolicious/Guides/Cookbook>, > Mojo::Server::Prefork preloads the application in the manager/master > process, and Hypnotoad is based off that, so that would help.) > > It does seem like changes to the OpenAPI plugin would be needed for caching. > > I'm going to try backporting your change and try pre-loading and see how far that gets me. > > David Cook > Software Engineer > Prosentient Systems > Suite 7.03 > 6a Glen St > Milsons Point NSW 2061 > Australia > > Office: 02 9212 0899 > Online: 02 8005 0595 > > -----Original Message----- > From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org>> On > Behalf Of Ere Maijala > Sent: Tuesday, 20 April 2021 4:48 PM > To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Optimizing Starman startup > > Hi, > > I did some work on improving it here: > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700> > > That shaved a good bit of time from it, but it's still a heavy > operation, and it would make sense to > > 1.) avoid doing it too often > > 2.) cache the results and avoid doing it if results are cached > > If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. > > --Ere > > dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15: >> Hi all, >> >> Do you despair when you see the following periodically in “top” when >> a starman worker is recreated ? >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> >> 9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> Or the following in top when you install koha-common package or >> restart the koha-common service? >> >> 11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> Well, I still have a lot of investigation left to do, but I notice 1 >> place that a lot of time taken is here (per worker): >> >> my $validator = JSON::Validator::OpenAPI::Mojolicious->new; >> >> $validator->load_and_validate_schema( >> >> $self->home->rel_file("api/v1/swagger/swagger.json"), >> >> { >> >> allow_invalid_ref => 1, >> >> } >> >> ); >> >> push @{$self->routes->namespaces}, 'Koha::Plugin'; >> >> my $spec = $validator->schema->data; >> >> $self->plugin( >> >> 'Koha::REST::Plugin::PluginRoutes' => { >> >> spec => $spec, >> >> validator => $validator >> >> } >> >> ); >> >> $self->plugin( >> >> OpenAPI => { >> >> spec => $spec, >> >> route => >> $self->routes->under('/api/v1')->to('Auth#under'), >> >> allow_invalid_ref => >> >> 1, # required by our spec because $ref directly >> under >> >> # Paths-, Parameters-, Definitions- & >> Info-object >> >> # is not allowed by the OpenAPI specification. >> >> } >> >> ); >> >> Anyone have ideas for improving this? Do we have to validate the >> schema every time? Can we move the schema validation into a different >> module and preload it into Starman using the -M flag so that it’s >> done >> 1 time per Starman master instance rather than 1 time per Starman worker instance? >> >> I find “/usr/share/koha/api/v1/app.pl <http://app.pl>” to be the bane of deployments, >> as it puts a massive load on a server, when you have multiple Koha >> instances on the server. >> >> David Cook >> >> Software Engineer >> >> Prosentient Systems >> >> Suite 7.03 >> >> 6a Glen St >> >> Milsons Point NSW 2061 >> >> Australia >> >> Office: 02 9212 0899 >> >> Online: 02 8005 0595 >> >> >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> >> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : >> https://git.koha-community.org/ <https://git.koha-community.org/> bugs : >> https://bugs.koha-community.org/ <https://bugs.koha-community.org/> >> > > -- > Ere Maijala > Kansalliskirjasto / The National Library of Finland > _______________________________________________ > Koha-devel mailing list > Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> > website : https://www.koha-community.org/ <https://www.koha-community.org/> git : > https://git.koha-community.org/ <https://git.koha-community.org/> bugs : > https://bugs.koha-community.org/ <https://bugs.koha-community.org/> > >
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : https://git.koha-community.org/ <https://git.koha-community.org/> bugs : https://bugs.koha-community.org/ <https://bugs.koha-community.org/>
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
But now that I think about it again... Jan's change is actually all we need. Within Koha, we could do the checksum calculation, and then skip the validation if we have a local record that our schema has already been validated. (I guess we'd set the cache after Mojoliciou::Plugin::OpenAPI successfully registers without dying from errors.) So we don't need to change JSON::Validator at all. We just need a Mojolicious::Plugin::OpenAPI that allows us to skip validation. We then handle the checksum and caching within our application. *mindblown* Now we just need for Jan to release the new version and for Koha to catch up to the point of being able to use that version hehe. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: dcook@prosentient.com.au <dcook@prosentient.com.au> Sent: Wednesday, 28 April 2021 2:30 PM To: 'Ere Maijala' <ere.maijala@helsinki.fi>; 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> Cc: 'Koha Devel' <koha-devel@lists.koha-community.org> Subject: RE: [Koha-devel] Optimizing Starman startup Based on my request, Jan has added an option to skip validation (https://github.com/jhthorsen/mojolicious-plugin-openapi/commit/673079d19f827...). However, he doesn't seem to want to implement my cache idea. He has said that he's willing to consider a PR, so I'll look at doing that some evening after work. He did propose freezing a JSON::Validator object and passing that between processes, but I don't really like that idea. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi> Sent: Tuesday, 27 April 2021 3:28 PM To: dcook@prosentient.com.au; 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> Cc: 'Koha Devel' <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Thanks for digging up all the information. I think adding a caching validator would provide a nice improvement, so I'm in favor of it, and if I were to decide, I'd also include it in Koha. Still, getting it upstream would be great in the long run, so I'd support that as well. --Ere dcook@prosentient.com.au kirjoitti 27.4.2021 klo 5.34:
After more experimenting, I have concluded that JSON::Validator->validate() is the culprit in terms of CPU time and memory usage.
Fortunately, I’ve determined that Mojo::JSON and Digest::MD5 can be used together to create consistent reproducible checksums, which could be used for caching validated schemas.
Of course, a solution would involve changes to JSON::Validator (and possibly Mojolicious::Plugin::OpenAPI depending on the chosen solution), and then we’d have to wait for the new and improved version to come downstream, so we wouldn’t see the benefit of this for years.
That said… we could always roll our own JSON::Validator. And if we don’t want to do it as a community, I could always just do it myself.
In terms of testing… with 18 CPUs I can restart 60 instances (120 processes) and get through the app setup in about 60 seconds with significant server load, when using validation. Without validation, I can do it in 20 seconds without significant server load (beyond a few short-lived CPU spikes).
I’m thinking about writing a patch and sending a pull request for JSON::Validator, but also really thinking about implementing it locally too at least in the meantime.
I haven’t heard from the author of JSON::Validator for a little while now, but I hope I do hear back from him. I think it would be a great addition to the library.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *dcook@prosentient.com.au *Sent:* Monday, 26 April 2021 7:01 PM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org> *Subject:* Re: [Koha-devel] Optimizing Starman startup
After some more experimenting, it’s clear that the problem isn’t JSON::Validator::OpenAPI::Mojolicious or Koha::REST::Plugin::PluginRoutes. If you exclude Mojolicious::Plugin::OpenAPI, the startup is very fast. It’s 30 seconds start to finish to restart 60 instances and each instance restarts very quickly.
When using Mojolicious::Plugin::OpenAPI, it takes about 3 minutes and there’s a fair bit of downtime during that time.
When I do a strace, I’m noticing that a process can spend 30 seconds just allocating memory for Mojolicious::Plugin::OpenAPI, but it only happens once you hit a certain volume of processes. If you’re just starting up 1 or 2, then it’s only a couple seconds. But if you have say 60-120 processes, it can take up to 30 seconds for Mojolicious::Plugin::OpenAPI to do its work. I’m putting 10 CPUs to this work, but clearly that’s not enough. I imagine there may be other bottlenecks accessing the memory as well.
Has anyone profiled Mojolicious before? I’m guessing maybe Martin?
I suspect that this is just a problem that I’m going to have to live with but maybe it is a case where I can find a way to optimize Mojolicious::Plugin::OpenAPI.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org>> *On Behalf Of *dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> *Sent:* Monday, 26 April 2021 5:12 PM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* Re: [Koha-devel] Optimizing Starman startup
So I just tried the following…
--
root@kohadevbox:koha(master)$ npm install -g swagger-cli
/usr/bin/swagger-cli -> /usr/lib/node_modules/swagger-cli/swagger-cli.js
npm WARN @apidevtools/swagger-parser@10.0.2 requires a peer of openapi-types@>=7 but none is installed. You must install peer dependencies yourself.
+ swagger-cli@4.0.4 <mailto:swagger-cli@4.0.4>
added 46 packages from 27 contributors in 8.203s
--
root@kohadevbox:koha(master)$ time swagger-cli bundle api/v1/swagger/swagger.json --outfile api/v1/swagger/openapi.json --type json
Created api/v1/swagger/openapi.json from api/v1/swagger/swagger.json
real 0m0.296s
user 0m0.346s
sys 0m0.032s
openapi.json is 10891 lines long but it actually contains 741 $ref lines like "$ref": "#/definitions/error" and "$ref": "#/definitions/patron_extended_attribute".
--
Now to do some benchmarking… I ran the following code:
#!/usr/bin/perl
use JSON::Validator::OpenAPI::Mojolicious;
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
my $spec = $validator->bundle({
replace => 1,
schema => "api/v1/swagger/swagger.json",
});
The first time I ran it… it took 1.343 seconds. The second time and subsequent times it took .354 seconds. (That’s using Ubuntu 20.04 and JSON Validator 3.14.) That suggests caching although I’m not sure where. I don’t see anything obvious in /usr/share/perl5/JSON/Validator/cache.
Trying with openapi.json yields .280 seconds instead of .354 seconds. It’s faster, but not significantly.
So that suggests that the problem is actually with Koha::REST::Plugin::PluginRoutes or Mojolicious::Plugin::OpenAPI more specifically…
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> *Sent:* Monday, 26 April 2021 11:24 AM *To:* 'Renvoize, Martin' <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Cc:* 'Koha Devel' <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* RE: [Koha-devel] Optimizing Starman startup
I think that I accidentally offended him, as he hasn’t responded to me since his initial response.
I do wonder if reducing the number of references would help, although I wonder how easy that would be to do in practice. It looks like we have about 5767 lines of JSON all up as is… so it would probably get even bigger if we dereferenced them.
Oh… here’s a thought. Why don’t we compile it? According to https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple -files/ <https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multipl e-files/>, you can maintain many different files, and then use something like swagger-cli to create a single built/compiled OpenAPI file.
That way JSON::Validator wouldn’t need to resolve any references for the core API. I don’t know if the plugins have any $ref in them but I’m guessing not (just based on Coverflow). So that could be a big win.
I’m working on other things at the moment, but I’m going to put that on my eternal list.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Renvoize, Martin <martin.renvoize@ptfs-europe.com <mailto:martin.renvoize@ptfs-europe.com>> *Sent:* Friday, 23 April 2021 5:24 PM *To:* David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> *Cc:* Koha Devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org>> *Subject:* Re: [Koha-devel] Optimizing Starman startup
Jan's code is certainly challenging to read and understand at times I agree.. I used to contribute to the plugin a number of years ago now.. but the project that gave me time to play with that has since been sold on so I'm not involved at the level I used to be.. he uses lots of Perl foo which often takes me a long time to wrap my head around.
As for the refs, I think we split our spec up too much in all honesty.. even the swagger spec suggest we went too far.. I think I might have been unclear when I first pushed for a split from one massive file. We could/should certainly reduce that somewhat.. it'll be interesting to see if it makes much difference.. that could be a fairly quick win.
On Thu, 22 Apr 2021, 12:32 am , <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> wrote:
Hi Ere,
I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins.
Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about.
At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS.
I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind.
Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment...
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Ere Maijala <ere.maijala@helsinki.fi <mailto:ere.maijala@helsinki.fi>> Sent: Wednesday, 21 April 2021 6:31 PM To: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>; koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup
Hi David,
I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it.
Thanks for taking a look!
--Ere
dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 21.4.2021 klo 3.28: > Hi Ere, > > Thanks for your reply. 24700 looks much better. I'll look at backporting it locally. > > Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/l... <https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm> and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement? > > Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky. > > At a glance, we might be able to pre-load the app into the Starman > master process pre-fork. There are warnings about doing that with open > database connections, so we'd need to review plack.psgi, but a quick > glance suggests it might be OK. (Alternatively, I have wondered about > running the REST API as a separate process apart from Starman using > hypnotoad. According to > https://docs.mojolicious.org/Mojolicious/Guides/Cookbook <https://docs.mojolicious.org/Mojolicious/Guides/Cookbook>, > Mojo::Server::Prefork preloads the application in the manager/master > process, and Hypnotoad is based off that, so that would help.) > > It does seem like changes to the OpenAPI plugin would be needed for caching. > > I'm going to try backporting your change and try pre-loading and see how far that gets me. > > David Cook > Software Engineer > Prosentient Systems > Suite 7.03 > 6a Glen St > Milsons Point NSW 2061 > Australia > > Office: 02 9212 0899 > Online: 02 8005 0595 > > -----Original Message----- > From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org>> On > Behalf Of Ere Maijala > Sent: Tuesday, 20 April 2021 4:48 PM > To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Optimizing Starman startup > > Hi, > > I did some work on improving it here: > > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700> > > That shaved a good bit of time from it, but it's still a heavy > operation, and it would make sense to > > 1.) avoid doing it too often > > 2.) cache the results and avoid doing it if results are cached > > If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching. > > --Ere > > dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> kirjoitti 20.4.2021 klo 6.15: >> Hi all, >> >> Do you despair when you see the following periodically in “top” when >> a starman worker is recreated ? >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> >> 9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> Or the following in top when you install koha-common package or >> restart the koha-common service? >> >> 11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> 11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 >> /usr/share/koha/api/v1/app.pl <http://app.pl> >> >> Well, I still have a lot of investigation left to do, but I notice 1 >> place that a lot of time taken is here (per worker): >> >> my $validator = JSON::Validator::OpenAPI::Mojolicious->new; >> >> $validator->load_and_validate_schema( >> >> $self->home->rel_file("api/v1/swagger/swagger.json"), >> >> { >> >> allow_invalid_ref => 1, >> >> } >> >> ); >> >> push @{$self->routes->namespaces}, 'Koha::Plugin'; >> >> my $spec = $validator->schema->data; >> >> $self->plugin( >> >> 'Koha::REST::Plugin::PluginRoutes' => { >> >> spec => $spec, >> >> validator => $validator >> >> } >> >> ); >> >> $self->plugin( >> >> OpenAPI => { >> >> spec => $spec, >> >> route => >> $self->routes->under('/api/v1')->to('Auth#under'), >> >> allow_invalid_ref => >> >> 1, # required by our spec because $ref directly >> under >> >> # Paths-, Parameters-, Definitions- & >> Info-object >> >> # is not allowed by the OpenAPI specification. >> >> } >> >> ); >> >> Anyone have ideas for improving this? Do we have to validate the >> schema every time? Can we move the schema validation into a different >> module and preload it into Starman using the -M flag so that it’s >> done >> 1 time per Starman master instance rather than 1 time per Starman worker instance? >> >> I find “/usr/share/koha/api/v1/app.pl <http://app.pl>” to be the bane of deployments, >> as it puts a massive load on a server, when you have multiple Koha >> instances on the server. >> >> David Cook >> >> Software Engineer >> >> Prosentient Systems >> >> Suite 7.03 >> >> 6a Glen St >> >> Milsons Point NSW 2061 >> >> Australia >> >> Office: 02 9212 0899 >> >> Online: 02 8005 0595 >> >> >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> >> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> >> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : >> https://git.koha-community.org/ <https://git.koha-community.org/> bugs : >> https://bugs.koha-community.org/ <https://bugs.koha-community.org/> >> > > -- > Ere Maijala > Kansalliskirjasto / The National Library of Finland > _______________________________________________ > Koha-devel mailing list > Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> > website : https://www.koha-community.org/ <https://www.koha-community.org/> git : > https://git.koha-community.org/ <https://git.koha-community.org/> bugs : > https://bugs.koha-community.org/ <https://bugs.koha-community.org/> > >
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : https://git.koha-community.org/ <https://git.koha-community.org/> bugs : https://bugs.koha-community.org/ <https://bugs.koha-community.org/>
-- Ere Maijala Kansalliskirjasto / The National Library of Finland
Hi :) On 21-04-28 06:40, dcook@prosentient.com.au wrote:
But now that I think about it again... Jan's change is actually all we need.
That's a good news :D So what are the perf results of this? IIUC for your use case (spawning 120 workers with a 10 core CPU) * 2 minutes 45 seconds before bug 24700, i.e. < 20.05 * 1 minute 30 seconds Koha >= 20.05 * 20 sec when validation could be disabled with a future version of Mojolicious::Plugin::OpenAPI -- Victor Grousset/tuxayo
There's a number of things that are improved. I should say this testing was done with a custom Koha 19.11. First is speed, so 20 seconds to (re)start 120 workers with 20 CPUs without validation instead of taking 2 min 45 seconds w/o 24700 patch and 1 min 30 seconds w/ 24700 patch with 10 CPUs. (I was messing around with CPU count, so I don't have perfect 1-to-1 tests for everything.) That's a start to finish of the first instance (re)started and the last Starman process having loaded app.pl and started listening for connections. The actual (re)start time for each individual process would be less. Second is server load. With or without 24700 patch, the server load is massive (re)starting 120 workers with 10 or 20 CPUs. The server spends a lot of CPU time allocating memory during the validation process, so the server gets CPU bound during the validation process, and your instances experience latency at best and downtime at worst. Without the validation, I couldn't detect any latency or downtime when doing a (re)start. My testing could've been flawed, but the startup time was fast enough that I didn't notice a blip. Well, the one exception is when doing a time-consuming task like a big search. If I was in the middle of a long search, obviously a restart cut that off as it severed the TCP connection. But that's unavoidable. Anyway, I hope that helps? As I mentioned before, we probably won't be able to skip the validation for years, unless we roll our own Mojolicious::Plugin::OpenAPI... David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Victor Grousset/tuxayo Sent: Thursday, 29 April 2021 3:51 AM To: 'Koha Devel' <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Optimizing Starman startup Hi :) On 21-04-28 06:40, dcook@prosentient.com.au wrote:
But now that I think about it again... Jan's change is actually all we need.
That's a good news :D So what are the perf results of this? IIUC for your use case (spawning 120 workers with a 10 core CPU) * 2 minutes 45 seconds before bug 24700, i.e. < 20.05 * 1 minute 30 seconds Koha >= 20.05 * 20 sec when validation could be disabled with a future version of Mojolicious::Plugin::OpenAPI -- Victor Grousset/tuxayo _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Great results! Thanks for pushing this :) On 21-04-29 02:00, dcook@prosentient.com.au wrote:
As I mentioned before, we probably won't be able to skip the validation for years, unless we roll our own Mojolicious::Plugin::OpenAPI...
Or install via CPAN for instances that are facing the issue? Cheers, -- Victor Grousset/tuxayo
participants (5)
-
dcook＠prosentient.com.au -
Ere Maijala -
Julian Maurice -
Renvoize, Martin -
Victor Grousset/tuxayo