http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |11395 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #1 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 35444 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=35444&action=edit Bug 13606: Remove previous job data from the session Bug 11395 uses the session to store the report sent to the interface (update on biblio XX OK, etc.). If the session is stored in mysql, the max size of the sessions.a_session will be reached easily (TEXT field). To reproduce: 0/ Set SessionStorage to 'mysql' 1/ Create a file with 500 biblionumbers: mysql -e "select biblionumber from biblio limit 500;" | tail -n 500 > /tmp/biblionumbers.txt 2/ Define a marc modification template (something like "delete field 99$9" is nice) 3/ Load the /tmp/biblionumbers.txt in the batch biblio modification tool 4/ Repeat 3 You will get: Syck parser (line 1534, column 6): syntax error at /usr/lib/i386-linux-gnu/perl5/5.20/YAML/Syck.pm line 75. None Koha page is reachable. It comes from get_template_and_user > checkauth > get_session > l.1595 $session = new CGI::Session("driver:MySQL;serializer:yaml;id:md5", $sessionID, {Handle=>$dbh}); 5/ Get your sessionID contained in your CGISESSID cookie Have a look at the value a_session in the sessions table: mysql> select a_session from sessions where id="YOUR_SESSIONID"; You should see that the yaml is not correctly ended. The size of the DB field has been reached and the yaml is truncated. Test plan: 0/ Delete your CGISESSID cookie 1/ Try to reproduce the previous issue 2/ After the second batch, have a look at the sessions table and confirm that only one job_$JOB_ID exist in the yaml -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10601 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #35444|0 |1 is obsolete| | --- Comment #4 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 37597 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37597&action=edit [SIGNED-OFF] Bug 13606: Remove previous job data from the session Bug 11395 uses the session to store the report sent to the interface (update on biblio XX OK, etc.). If the session is stored in mysql, the max size of the sessions.a_session will be reached easily (TEXT field). To reproduce: 0/ Set SessionStorage to 'mysql' 1/ Create a file with 500 biblionumbers: mysql -e "select biblionumber from biblio limit 500;" | tail -n 500 > /tmp/biblionumbers.txt 2/ Define a marc modification template (something like "delete field 99$9" is nice) 3/ Load the /tmp/biblionumbers.txt in the batch biblio modification tool 4/ Repeat 3 You will get: Syck parser (line 1534, column 6): syntax error at /usr/lib/i386-linux-gnu/perl5/5.20/YAML/Syck.pm line 75. None Koha page is reachable. It comes from get_template_and_user > checkauth > get_session > l.1595 $session = new CGI::Session("driver:MySQL;serializer:yaml;id:md5", $sessionID, {Handle=>$dbh}); 5/ Get your sessionID contained in your CGISESSID cookie Have a look at the value a_session in the sessions table: mysql> select a_session from sessions where id="YOUR_SESSIONID"; You should see that the yaml is not correctly ended. The size of the DB field has been reached and the yaml is truncated. Test plan: 0/ Delete your CGISESSID cookie 1/ Try to reproduce the previous issue 2/ After the second batch, have a look at the sessions table and confirm that only one job_$JOB_ID exist in the yaml Signed-off-by: Frederic Demians <f.demians@tamil.fr> I can reproduce de bug. I can see also the useless C4::BackgroundJob session data. I confirm that modifying two consecutive 500 records batches broke entirely the session YAML representation deserialization. And finally I confirm that this patch fixes all those issues. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major --- Comment #6 from Jonathan Druart <jonathan.druart@biblibre.com> --- I am raising the priority of this one, it should absolutely be present in 3.20. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37601|0 |1 is obsolete| | --- Comment #8 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 37621 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37621&action=edit Bug 13606: Upgrade sessions.a_session to MEDIUMTEXT The previous patch fixed partially the problem. If you try to update more that 1000 records, you will get the same error. This patch change the type of sessions.a_session from TEXT (64kb) MEDIUMTEXT (32mb). Test plan: 1/ Try to modify a huge amount of records (1000 should be enough) 2/ Confirm it worked! Signed-off-by: Frederic Demians <f.demians@tamil.fr> I confirm that with more than 1000 records the session object is truncated, and then can't be anymore be desirialized, blocking completely Koha. Fixed with this patch. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #9 from Frédéric Demians <frederic@tamil.fr> --- (In reply to Jonathan Druart from comment #7)

Actually I am wondering what happens if 2 jobs are launched simultaneously :-/

Good question! Above ~1000 records, the processing time is long, and the UI feedback is not that good. So users could be tempted to refresh the page. It may be advisable to limit processing to let say 1000 records. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #11 from Jonathan Druart <jonathan.druart@biblibre.com> --- The first patch should be improved to only delete data from finished jobs. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37597|0 |1 is obsolete| | --- Comment #12 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 37627 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37627&action=edit Bug 13606: Remove previous job data from the session -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtompset@hotmail.com --- Comment #14 from M. Tompsett <mtompset@hotmail.com> --- Why is there no "Started: 2015-04-09 12:58:00" type value in the job session information? You could easily also delete anything that is more than 24 hours old, if there was a way to determine it. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #16 from M. Tompsett <mtompset@hotmail.com> --- I successfully (and that's not a good thing) ran two batch modify concurrently, and one progress bar hung, but the other finished. I suspect the process was killed by OS, because the javascript kept pulling the 99% just fine. However, on a positive note, this does make a significant improvement. In the same kind of test ran multiple times, I actually had the error (mentioned in comment #1) triggered behind the scenes and the progress bar seemed to get stuck before the patch. I did not encounter the same problem after. Though, the patch for the paragraph 1 problem would likely solve, as a side-effect the hidden hang in the paragraph 2 problem. If we store the process ID of the child in the job session, then the progress javascript call can check for the existence of that process if the status is not completed. If the process does not exist, then it can change the status to failed OR perhaps attempt to restart with a smaller data set in order to continue. This, of course, is discussion for the creation of another bug report. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37621|0 |1 is obsolete| | --- Comment #17 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 37633 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37633&action=edit Bug 13606: Upgrade sessions.a_session to MEDIUMTEXT The previous patch fixed partially the problem. If you try to update more that 1000 records, you will get the same error. This patch change the type of sessions.a_session from TEXT (64kb) MEDIUMTEXT (32mb). Test plan: 1/ Try to modify a huge amount of records (1000 should be enough) 2/ Confirm it worked! Signed-off-by: Frederic Demians <f.demians@tamil.fr> I confirm that with more than 1000 records the session object is truncated, and then can't be anymore be desirialized, blocking completely Koha. Fixed with this patch. Signed-off-by: Mark Tompsett <mtompset@hotmail.com> This only pushes off the problem expontentially (64KB vs. 16MB), but this is a good start. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37627|0 |1 is obsolete| | --- Comment #19 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 37635 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37635&action=edit Bug 13606: Remove previous job data from the session Bug 11395 uses the session to store the report sent to the interface (update on biblio XX OK, etc.). If the session is stored in mysql, the max size of the sessions.a_session will be reached easily (TEXT field). To reproduce: 0/ Set SessionStorage to 'mysql' 1/ Create a file with 500 biblionumbers: mysql -e "select biblionumber from biblio limit 500;" | tail -n 500 > /tmp/biblionumbers.txt 2/ Define a marc modification template (something like "delete field 99$9" is nice) 3/ Load the /tmp/biblionumbers.txt in the batch biblio modification tool 4/ Repeat 3 You will get: Syck parser (line 1534, column 6): syntax error at /usr/lib/i386-linux-gnu/perl5/5.20/YAML/Syck.pm line 75. None Koha page is reachable. It comes from get_template_and_user > checkauth > get_session > l.1595 $session = new CGI::Session("driver:MySQL;serializer:yaml;id:md5", $sessionID, {Handle=>$dbh}); 5/ Get your sessionID contained in your CGISESSID cookie Have a look at the value a_session in the sessions table: mysql> select a_session from sessions where id="YOUR_SESSIONID"; You should see that the yaml is not correctly ended. The size of the DB field has been reached and the yaml is truncated. Test plan: 0/ Delete your CGISESSID cookie 1/ Try to reproduce the previous issue 2/ After the second batch, have a look at the sessions table and confirm that only one job_$JOB_ID exist in the yaml Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Confirmed that last completed run is the only thing in the session, under normal circumstances, now. This will go a long way to preventing the 16MB MEDIUMTEXT from filling up. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #20 from Frédéric Demians <frederic@tamil.fr> --- Jonathan, reading your code, I understand better now how it interact with C4::Background module. Your last patch clear the batch modification job data when a new job is created. Wouldn't it be simpler to clear it just after the job is completed in batch_record_modification.pl? I will do a proposal patch for that, hoping I don't miss something important. And, yes, I confirm that progress bar is also broken in batch item modification. It may works that way since a while. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37636|application/mbox |text/plain mime type| | Attachment #37636|0 |1 is patch| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #23 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to Frédéric Demians from comment #20)

Jonathan, reading your code, I understand better now how it interact with C4::Background module. Your last patch clear the batch modification job data when a new job is created. Wouldn't it be simpler to clear it just after the job is completed in batch_record_modification.pl? I will do a proposal patch for that, hoping I don't miss something important.

It works nicely. I have tried to add this line in the finish method, but did not work. The only concern could be in case we need debugging. If something wrong happens it is good to know what kind of batch has been launched. Apart from that my patch was not much better than yours...

And, yes, I confirm that progress bar is also broken in batch item modification. It may works that way since a while.

That's really weird, I'm sure it worked when I have submitted my patch set on bug 11395. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #25 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to Jonathan Druart from comment #23)

...

And, yes, I confirm that progress bar is also broken in batch item modification. It may works that way since a while.

That's really weird, I'm sure it worked when I have submitted my patch set on bug 11395.

It does not work on my dev install, but works perfectly on sandboxes (tested with batch records modification, and import). I would suspect the perl version (5.20.1 vs 5.14.2). -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #26 from Frédéric Demians <frederic@tamil.fr> --- (In reply to Jonathan Druart from comment #23)

I have tried to add this line in the finish method, but did not work.

As I understand background process, the finish() method is called when the job is achieved. Then the job result is stored in the session. And from there, the result can be retrieved by a web page (like batch modification page). So the job data clearing must separated from marking a job as finished. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #28 from Frédéric Demians <frederic@tamil.fr> --- (In reply to M. Tompsett from comment #27)

If you add a function to a C4 library, you need to add corresponding tests in a .t file too.

Is there somewhere a form to fill out in three copies? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #29 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 37648 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37648&action=edit Bug 13606 New test for added method to C4::BackgroundJob -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #31 from Jonathan Druart <jonathan.druart@biblibre.com> --- I have a patch for tests, will be ready soon. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37648|0 |1 is obsolete| | --- Comment #32 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 37658 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37658&action=edit Bug 13606: Add tests and fix existing ones Some tests was wrong and not executed in a transaction. Lot of sessions were created in DB because the session was never flushed. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #34 from Frédéric Demians <frederic@tamil.fr> --- Jonathan: +1 for testing correctly Mark: +1 for keeping the quality high, or elevating it even -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37637|0 |1 is obsolete| | --- Comment #36 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 37663 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37663&action=edit Bug 13606 Clear job data in session when the job is completed Bug 11395 uses the session to store the report sent to the interface (update on biblio XX OK, etc.). If the session is stored in mysql, the max size of the sessions.a_session will be reached easily (TEXT field). To reproduce: 0/ Set SessionStorage to 'mysql' 1/ Create a file with 500 biblionumbers: mysql -e "select biblionumber from biblio limit 500;" | tail -n 500 > /tmp/biblionumbers.txt 2/ Define a marc modification template (something like "delete field 99$9" is nice) 3/ Load the /tmp/biblionumbers.txt in the batch biblio modification tool 4/ Repeat 3 You will get: Syck parser (line 1534, column 6): syntax error at /usr/lib/i386-linux-gnu/perl5/5.20/YAML/Syck.pm line 75. None Koha page is reachable. It comes from get_template_and_user > checkauth > get_session > l.1595 $session = new CGI::Session("driver:MySQL;serializer:yaml;id:md5", $sessionID, {Handle=>$dbh}); 5/ Get your sessionID contained in your CGISESSID cookie Have a look at the value a_session in the sessions table: mysql> select a_session from sessions where id="YOUR_SESSIONID"; You should see that the yaml is not correctly ended. The size of the DB field has been reached and the yaml is truncated. Test plan: 0/ Delete your CGISESSID cookie 1/ Try to reproduce the previous issue 2/ After the second batch, have a look at the sessions table and confirm that only one job_$JOB_ID exist in the yaml Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> NOTE: Nicely clears batch job session information. (2) is incorrect. This cleans up after a full run. Signed-off-by: Mark Tompsett <mtompset@hotmail.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #38 from M. Tompsett <mtompset@hotmail.com> --- This is so ready for QA now. :) Thanks everyone. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #37662|0 |1 is obsolete| | Attachment #37663|0 |1 is obsolete| | Attachment #37664|0 |1 is obsolete| | --- Comment #39 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 38058 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38058&action=edit [PASSED QA] Bug 13606: Upgrade sessions.a_session to MEDIUMTEXT The previous patch fixed partially the problem. If you try to update more that 1000 records, you will get the same error. This patch change the type of sessions.a_session from TEXT (64kb) MEDIUMTEXT (32mb). Test plan: 1/ Try to modify a huge amount of records (1000 should be enough) 2/ Confirm it worked! Signed-off-by: Frederic Demians <f.demians@tamil.fr> I confirm that with more than 1000 records the session object is truncated, and then can't be anymore be desirialized, blocking completely Koha. Fixed with this patch. Signed-off-by: Mark Tompsett <mtompset@hotmail.com> This only pushes off the problem expontentially (64KB vs. 16MB), but this is a good start. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #41 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 38060 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38060&action=edit [PASSED QA] Bug 13606 Clear job data in session when the job is completed Bug 11395 uses the session to store the report sent to the interface (update on biblio XX OK, etc.). If the session is stored in mysql, the max size of the sessions.a_session will be reached easily (TEXT field). To reproduce: 0/ Set SessionStorage to 'mysql' 1/ Create a file with 500 biblionumbers: mysql -e "select biblionumber from biblio limit 500;" | tail -n 500 > /tmp/biblionumbers.txt 2/ Define a marc modification template (something like "delete field 99$9" is nice) 3/ Load the /tmp/biblionumbers.txt in the batch biblio modification tool 4/ Repeat 3 You will get: Syck parser (line 1534, column 6): syntax error at /usr/lib/i386-linux-gnu/perl5/5.20/YAML/Syck.pm line 75. None Koha page is reachable. It comes from get_template_and_user > checkauth > get_session > l.1595 $session = new CGI::Session("driver:MySQL;serializer:yaml;id:md5", $sessionID, {Handle=>$dbh}); 5/ Get your sessionID contained in your CGISESSID cookie Have a look at the value a_session in the sessions table: mysql> select a_session from sessions where id="YOUR_SESSIONID"; You should see that the yaml is not correctly ended. The size of the DB field has been reached and the yaml is truncated. Test plan: 0/ Delete your CGISESSID cookie 1/ Try to reproduce the previous issue 2/ After the second batch, have a look at the sessions table and confirm that only one job_$JOB_ID exist in the yaml Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> NOTE: Nicely clears batch job session information. (2) is incorrect. This cleans up after a full run. Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #43 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 38431 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38431&action=edit Bug 13606: (RM followup) DBIx updates Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |chris@bigballofwax.co.nz --- Comment #45 from Chris Cormack <chris@bigballofwax.co.nz> --- Doesn't apply and I think is not needed on 3.18.x -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13606 --- Comment #46 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to Chris Cormack from comment #45)

Doesn't apply and I think is not needed on 3.18.x

No it's not, bug 11395 will be in 3.20 -- You are receiving this mail because: You are watching all bug changes.