[Bug 42596] New: Add ability for reports APIs ( svc/report, opac/svc/report ) to run reports as background jobs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Bug ID: 42596 Summary: Add ability for reports APIs ( svc/report, opac/svc/report ) to run reports as background jobs Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Reports Assignee: koha-bugs@lists.koha-community.org Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org CC: lisette@bywatersolutions.com Right now, all svc reports act as "foreground" jobs and return data immediately. This can cause issues such as apache timeouts for long running jobs. It would make sense to have the ability for a caller to ask for a background job and be given an id so that it can a) look up the status of the job ( waiting, running, finished, failed ) and if the job is finished, b) download the results of the report. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |41921 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41921 [Bug 41921] Add ability to run reports as background jobs -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #1 from David Cook <dcook@prosentient.com.au> --- This could be an opportunity to look at permissions with the SVC report APIs as well as currently they do act as a way around Koha permissions... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |42108 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42108 [Bug 42108] Add ability to specify that a report can only be run as a foreground or background report -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|41921 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41921 [Bug 41921] Add ability to run reports as background jobs -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #2 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199698 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199698&action=edit Bug 42596: Add ability for reports APIs ( svc/report, opac/svc/report ) to run reports as background jobs Right now, all svc reports act as foreground jobs and return data immediately. This can cause issues such as apache timeouts for long running jobs. It would make sense to have the ability for a caller to ask for a background job and be given an id so that it can a) look up the status of the job ( waiting, running, finished, failed ) and if the job is finished, b) download the results of the report. This patch adds a "run_type" query parameter to svc/report and opac/svc/report. When run_type is set to background the endpoint enqueues a background job report and returns a 202 Accepted with the URL the caller polls for status With the no run_type ( or a value of "foreground" ) the APIs behave just as they always have. Jobs that do not exist return 410 Gone. Request that contravene the report run type get 422 Unprocessable Entity. Staff can only poll their own background jobs via this API ( not sure if this is really needed, but it seemed good to have ). Example responses against svc/report: GET /cgi-bin/koha/svc/report?id=42&run_type=background -> 202 Accepted Location: /cgi-bin/koha/svc/report?job_id=99 {"job_id":99,"status":"new","status_url":"/cgi-bin/koha/svc/report?job_id=99"} # Running report GET /cgi-bin/koha/svc/report?job_id=99 -> 200 OK {"job_id":99,"status":"started"} # Finished report GET /cgi-bin/koha/svc/report?job_id=99 -> 200 OK {"job_id":99,"status":"finished","result_url":"/cgi-bin/koha/svc/report?saved_report_id=42"} # Report results GET /cgi-bin/koha/svc/report?saved_report_id=42 -> 200 OK [[1,2,3]] # Report results with the nice key/value pairs GET /cgi-bin/koha/svc/report?saved_report_id=42&annotated=1 -> 200 OK [{"n":1,"m":2,"p":3}] # Invalid job id GET /cgi-bin/koha/svc/report?job_id=99999 -> 410 Gone {"error":"job_not_found"} # Invalid report results retrieval id GET /cgi-bin/koha/svc/report?saved_report_id=99999 -> 410 Gone {"error":"saved_report_not_found"} # Run type request that is against the report's set run_type GET /cgi-bin/koha/svc/report?id=42&run_type=background -> 422 Unprocessable Entity {"error":"run_type_not_allowed","allowed":"foreground"} Test Plan: 1) Apply this patch 2) Restart all the things! 3) Create two public SQL reports: SELECT 1 AS n, 2 AS m, 3 AS p SELECT SLEEP(15) AS done 4) Browse to svc/report for the first report /cgi-bin/koha/svc/report?id=$REPORTID&login_userid=koha&login_password=koha 5) Note you get the rows array back 6) Tray again in background mode: /cgi-bin/koha/svc/report?id=$REPORTID&run_type=background&login_userid=koha&login_password=koha 7) Note the 202 Accepted, with status_url in the JSON body and a matching Location header 8) Poll the returned status_url: /cgi-bin/koha/svc/report?job_id=$REPORTID&login_userid=koha&login_password=koha 9) Note the envelope { job_id, status: 'finished', result_url } with no inline rows! 10) Get the result_url, the data should look just like a foreground run /cgi-bin/koha/svc/report?saved_report_id=$REPORTID&login_userid=koha&login_password=koha 11) Get the result_url with annotated=1 in the params, it should look like the foreground version that also as annotated=1 /cgi-bin/koha/svc/report?saved_report_id=$REPORTID&login_userid=koha&login_password=koha&annotated=1 12) Enqueue the slow report in background mode, capture the new job id: curl '/cgi-bin/koha/svc/report?id=$SLEEPY_REPORTID&run_type=background&login_userid=koha&login_password=koha' 13) Browse to its' status_url, note status "started" ( unless your jobs queue is backed up ) 14) Wait 16 seconds, reload the url, status should now be "finished" 17) Repeat 4-16 for the OPAC -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #3 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199699 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199699&action=edit Bug 42596: Return 404 in svc/report when report does not exist Patch from commit 2f5a488 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #4 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199700 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199700&action=edit Bug 42596: Use reports permissions instead of catalogue permission Patch from commit 47c0bfc -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #5 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199701 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199701&action=edit Bug 42596: Tidy svc/report Patch from commit c7811d6 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199698|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199699|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199700|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199701|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #6 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199702 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199702&action=edit Bug 42596: Add ability for reports APIs ( svc/report, opac/svc/report ) to run reports as background jobs Right now, all svc reports act as foreground jobs and return data immediately. This can cause issues such as apache timeouts for long running jobs. It would make sense to have the ability for a caller to ask for a background job and be given an id so that it can a) look up the status of the job ( waiting, running, finished, failed ) and if the job is finished, b) download the results of the report. This patch adds a "run_type" query parameter to svc/report and opac/svc/report. When run_type is set to background the endpoint enqueues a background job report and returns a 202 Accepted with the URL the caller polls for status With the no run_type ( or a value of "foreground" ) the APIs behave just as they always have. Jobs that do not exist return 410 Gone. Request that contravene the report run type get 422 Unprocessable Entity. Staff can only poll their own background jobs via this API ( not sure if this is really needed, but it seemed good to have ). Example responses against svc/report: GET /cgi-bin/koha/svc/report?id=42&run_type=background -> 202 Accepted Location: /cgi-bin/koha/svc/report?job_id=99 {"job_id":99,"status":"new","status_url":"/cgi-bin/koha/svc/report?job_id=99"} # Running report GET /cgi-bin/koha/svc/report?job_id=99 -> 200 OK {"job_id":99,"status":"started"} # Finished report GET /cgi-bin/koha/svc/report?job_id=99 -> 200 OK {"job_id":99,"status":"finished","result_url":"/cgi-bin/koha/svc/report?saved_report_id=42"} # Report results GET /cgi-bin/koha/svc/report?saved_report_id=42 -> 200 OK [[1,2,3]] # Report results with the nice key/value pairs GET /cgi-bin/koha/svc/report?saved_report_id=42&annotated=1 -> 200 OK [{"n":1,"m":2,"p":3}] # Invalid job id GET /cgi-bin/koha/svc/report?job_id=99999 -> 410 Gone {"error":"job_not_found"} # Invalid report results retrieval id GET /cgi-bin/koha/svc/report?saved_report_id=99999 -> 410 Gone {"error":"saved_report_not_found"} # Run type request that is against the report's set run_type GET /cgi-bin/koha/svc/report?id=42&run_type=background -> 422 Unprocessable Entity {"error":"run_type_not_allowed","allowed":"foreground"} Test Plan: 1) Apply this patch 2) Restart all the things! 3) Create two public SQL reports: SELECT 1 AS n, 2 AS m, 3 AS p SELECT SLEEP(15) AS done 4) Browse to svc/report for the first report /cgi-bin/koha/svc/report?id=$REPORTID&login_userid=koha&login_password=koha 5) Note you get the rows array back 6) Tray again in background mode: /cgi-bin/koha/svc/report?id=$REPORTID&run_type=background&login_userid=koha&login_password=koha 7) Note the 202 Accepted, with status_url in the JSON body and a matching Location header 8) Poll the returned status_url: /cgi-bin/koha/svc/report?job_id=$REPORTID&login_userid=koha&login_password=koha 9) Note the envelope { job_id, status: 'finished', result_url } with no inline rows! 10) Get the result_url, the data should look just like a foreground run /cgi-bin/koha/svc/report?saved_report_id=$REPORTID&login_userid=koha&login_password=koha 11) Get the result_url with annotated=1 in the params, it should look like the foreground version that also as annotated=1 /cgi-bin/koha/svc/report?saved_report_id=$REPORTID&login_userid=koha&login_password=koha&annotated=1 12) Enqueue the slow report in background mode, capture the new job id: curl '/cgi-bin/koha/svc/report?id=$SLEEPY_REPORTID&run_type=background&login_userid=koha&login_password=koha' 13) Browse to its' status_url, note status "started" ( unless your jobs queue is backed up ) 14) Wait 16 seconds, reload the url, status should now be "finished" 17) Repeat 4-16 for the OPAC -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #7 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199703 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199703&action=edit Bug 42596: Return 404 in svc/report when report does not exist Patch from commit 1fdb7ab -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #8 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199704 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199704&action=edit Bug 42596: Use reports permissions instead of catalogue permission Patch from commit 3722ea3 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 --- Comment #9 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199705 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199705&action=edit Bug 42596: Tidy svc/report Patch from commit ca2676c -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42720 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42720 [Bug 42720] Add REST API equivalents for svc/report and opac/svc/report -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #10 from David Nind <david@davidnind.com> --- Dependent bug, bug 42108, no longer applies 8-(... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@openfifth.c |y.org |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42596 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #11 from David Nind <david@davidnind.com> --- (In reply to David Nind from comment #10)
Dependent bug, bug 42108, no longer applies 8-(...
Bug 42108 (and dependencies) now apply. However, the patches for this bug no longer apply 8-(... ... Applying 4 patch(es): Patch application failed for attachment 199702 - Bug 42596: Add ability for reports APIs ( svc/report, opac/svc/report ) to run reports as background jobs ... -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org