[Bug 27421] New: Porting tools/stage-marc-import.pl to BackgroundJob
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Bug ID: 27421 Summary: Porting tools/stage-marc-import.pl to BackgroundJob Change sponsored?: --- Product: Koha Version: 20.11 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: MARC Bibliographic record staging/import Assignee: koha-bugs@lists.koha-community.org Reporter: domm@plix.at QA Contact: testopia@bugs.koha-community.org Hi! I recently started to convert tools/stage-marc-import.pl to use the new BackgroundJobs, because we're using the Plack version, which does not like the old forking implementation. While I've realized now that we can still use the old CGI script by running it directly via the frontend Apache, I'm not very happy with this workaround. So, here is my current first sort-of-working prototype: https://github.com/Koha-Community/Koha/compare/master...domm:MARCImporter Before I continue my work, I'd really appreciate some feedback if what I'm doing makes sense and matches your general plans. Obviously there is still a lot missing: * Proper feedback after starting the staging process (i.e. a link to the job) * Probably I will also need to check if the job has finished, and present a link to the next step (copy the staged records into the library) (which AFAIK will also need to be converted to BackgroundJobs) I also have a few questions: * Is it neccessary to get the job-size before starting it? If yes, then I assume I'll have to first parse the MARC file to get the job size, throwing the results away (and this might not work, because we want to do the parsing in the background in the first place). Or is there an easy way to get the number of records? * When I try to update job_size later, I get an exception (something like "job_size not tested") * Can I actually use BatchStageMarcRecords in BackgroundJob? * I see I can pass a batch size to BatchStageMarcRecords. But this will not result in more forking? This is "only" used to fire the progress callback? I'm also not sure if the bug tracker or the dev mailing list is the best place to discuss my questions? Anyway, looking forward to some feedback! Greetings, domm -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|20.11 |unspecified Assignee|koha-bugs@lists.koha-commun |domm@plix.at |ity.org | CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Yes, that's the idea. (In reply to Thomas Klausner from comment #0)
* Is it neccessary to get the job-size before starting it? If yes, then I assume I'll have to first parse the MARC file to get the job size, throwing the results away (and this might not work, because we want to do the parsing in the background in the first place). Or is there an easy way to get the number of records?
I had the same problem with bug 27344. If it get too complicated (or cause perf issues) I'd say we can consider we are processing 1 file and so its size would be 1 :) At least for a first step.
* When I try to update job_size later, I get an exception (something like "job_size not tested")
The accessor is ->size
* Can I actually use BatchStageMarcRecords in BackgroundJob?
Why not? Did you face a problem?
* I see I can pass a batch size to BatchStageMarcRecords. But this will not result in more forking? This is "only" used to fire the progress callback?
I don't understand the question. Isn't BatchStageMarcRecords used to import the record? No fork will be done if you are using Koha::BackgroundJob.
I'm also not sure if the bug tracker or the dev mailing list is the best place to discuss my questions?
Here it's perfect :) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #2 from David Cook <dcook@prosentient.com.au> ---
(In reply to Thomas Klausner from comment #0)
* Is it neccessary to get the job-size before starting it? If yes, then I assume I'll have to first parse the MARC file to get the job size, throwing the results away (and this might not work, because we want to do the parsing in the background in the first place). Or is there an easy way to get the number of records?
I had the same problem with bug 27344. If it get too complicated (or cause perf issues) I'd say we can consider we are processing 1 file and so its size would be 1 :) At least for a first step.
I'd say use a job size of 1 since it's 1 file. As you say, Thomas, I think that it's unreasonable to parse the MARC file just to get the number of records. I think that "tools/stage-marc-import.pl" should just be responsible for uploading the file, enqueuing the background job, and then either showing/referring to a job management view to review the results. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #3 from Thomas Klausner <domm@plix.at> --- (In reply to Jonathan Druart from comment #1)
* Can I actually use BatchStageMarcRecords in BackgroundJob?
Why not? Did you face a problem?
No, it's working. I just wasn't sure if this code might also need some work, because it's API feels a bit "old" (exported from C4::ImportBatch, long list of positional args..)
* I see I can pass a batch size to BatchStageMarcRecords. But this will not result in more forking? This is "only" used to fire the progress callback?
I don't understand the question. Isn't BatchStageMarcRecords used to import the record? No fork will be done if you are using Koha::BackgroundJob.
It helps to actually read the code before asking questions. (Well, it also helps to read it *after* asking..). I wasn't sure if maybe BatchStageMarcRecords would start a few seperate processes, passing each a number of elements. But $progress_interval is "just" used to count the progress and potentially fire the progress_callback. So all is good :-) Thanks for the feedback, I will now continue here... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |22417 See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=28128 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 [Bug 22417] Add a task queue -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Hi Thomas, any news from this? Is it ready for testing? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #5 from Thomas Klausner <domm@plix.at> --- Hi! No, it's not ready for testing yet, mostly because I realized that I also need to port the second step of the import process; plus, I was distracted by other projects... I'll try to finish this feature this month. Or do you have any sooner (release) deadlines? Greetings, domm -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #6 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I don't think it could be ready (Passed QA) before the 21.05 feature freeze. But I would love to have it for the beginning of the next release if possible :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Bug 27421 depends on bug 22417, which changed state. Bug 22417 Summary: Add a task queue https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #7 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Do you think you will have time to get back to this, Thomas? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Excited to hear more about this as well. I'm keen to see MARC imports happening as background jobs. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #9 from Thomas Klausner <domm@plix.at> --- We ended up using the commandline tool (bulkmarkimport.pl) to migrate the data. So I stopped working on porting the web tool to background jobs. I could give it another go, but I'll most likely will have to do that on my own time (which is fine by me regarding money etc, but it's the most limited time slot I have...) So yes, I'd like to finish this feature, but I'm not sure I can prepare a patch that's nice enough for you and/or easy to test (mostly because I still did not get around to set up a proper Koha dev environment). I also have some architectural concerns / ideas about how the background jobs are implemented (eg having to list all the background job classes in the source code seems a bit redundant), but I think it's better to discuss those in another issue (or just via mail) instead of bloating this issue. Greetings, domm -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Thomas Klausner from comment #9)
I could give it another go, but I'll most likely will have to do that on my own time (which is fine by me regarding money etc, but it's the most limited time slot I have...)
I can relate to that on several projects :(
So yes, I'd like to finish this feature, but I'm not sure I can prepare a patch that's nice enough for you and/or easy to test (mostly because I still did not get around to set up a proper Koha dev environment).
Also relatable. Regarding a proper Koha dev environment, take a look at https://gitlab.com/koha-community/koha-testing-docker. I think a lot of people run it on Mac/Linux but I run it on Windows. So happy to answer any questions you might have about it. It makes Koha dev much easier.
I also have some architectural concerns / ideas about how the background jobs are implemented (eg having to list all the background job classes in the source code seems a bit redundant), but I think it's better to discuss those in another issue (or just via mail) instead of bloating this issue.
I have voiced the same concern but I think we might be the minority there. I think Bug 29149 re-organized the hard-coded list into a different hard-coded list in Koha::BackgroundJob. However, Bug 27783 removed the hard-coded list from misc/background_jobs_worker.pl at least. So halfway there. I don't understand the resistance to not having a hard-coded allow list. I think its security value is very limited and just makes the system much less usable... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|domm@plix.at |jonathan.druart+koha@gmail. | |com Status|NEW |ASSIGNED --- Comment #11 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I am going to work on it this week. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #12 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 136127 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136127&action=edit Bug 27421: Use Background job for staging MARC records for import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #13 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 136128 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136128&action=edit Bug 27421: Commit and revert -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #14 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 136129 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136129&action=edit Bug 27421: Enqueue only one job for indexing Might need to be backported. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #15 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- With those 3 patches I think most of the work is done. To continue I need people to commit to an involvement on these patches. To start I would like someone familiar with the import process to test the workflow and see if they are happy. Then we will need to add test coverage and certainly provide some bug fixes. Who's onboard? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |In Discussion -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |30822 --- Comment #16 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Jonathan Druart from comment #14)
Created attachment 136129 [details] [review] Bug 27421: Enqueue only one job for indexing
Might need to be backported.
This is actually bug 30822. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30822 [Bug 30822] BatchCommit does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136129|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #15)
With those 3 patches I think most of the work is done.
To continue I need people to commit to an involvement on these patches. To start I would like someone familiar with the import process to test the workflow and see if they are happy.
Then we will need to add test coverage and certainly provide some bug fixes.
Who's onboard?
I'm keen. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #17)
(In reply to Jonathan Druart from comment #15)
Who's onboard?
I'm keen.
It looks like bug 30822 doesn't apply to master anymore. But 27421 cleanly applies without it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Failed QA --- Comment #19 from David Cook <dcook@prosentient.com.au> --- My test plan: 0. Apply patches 1. Comment out the following lines in /etc/koha/apache-shared-intranet-plack.conf: ProxyPass "/cgi-bin/koha/tools/manage-marc-import.pl" "!" ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!" 2. service apache2 reload 3. koha-plack --restart kohadev 4. Go to http://localhost:8081/cgi-bin/koha/catalogue/export.pl?format=marcxml&op=export&bib=29 5. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 6. Upload bib-29.marcxml file 7. Change "How to process items" to "Ignore items" 8. Click "Stage for import" 9. Click "View detail of the enqueued job" 10. Note that a job has been created in Koha at http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1 However, the job isn't getting processed. Note the following in /var/log/koha/kohadev/worker-output.log: Exception 'Koha::Exception' thrown 'stage_marc_for_import is not a valid job_type' -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |In Discussion --- Comment #20 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #19)
However, the job isn't getting processed. Note the following in /var/log/koha/kohadev/worker-output.log:
Exception 'Koha::Exception' thrown 'stage_marc_for_import is not a valid job_type'
I imagine that's because I need to restart the workers actually... I'll change my test plan... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136127|0 |1 is obsolete| | --- Comment #21 from David Cook <dcook@prosentient.com.au> --- Created attachment 136216 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136216&action=edit Bug 27421: Use Background job for staging MARC records for import Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136128|0 |1 is obsolete| | --- Comment #22 from David Cook <dcook@prosentient.com.au> --- Created attachment 136217 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136217&action=edit Bug 27421: Commit and revert Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #23 from David Cook <dcook@prosentient.com.au> --- My test plan: 0. Apply patches 1. Comment out the following lines in /etc/koha/apache-shared-intranet-plack.conf: ProxyPass "/cgi-bin/koha/tools/manage-marc-import.pl" "!" ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!" 2. service apache2 reload 3. service koha-common restart #NOTE: We need to restart starman/Plack and the koha workers 4. Go to http://localhost:8081/cgi-bin/koha/catalogue/export.pl?format=marcxml&op=export&bib=29 5. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 6. Upload bib-29.marcxml file 7. Change "How to process items" to "Ignore items" 8. Click "Stage for import" 9. Click "View detail of the enqueued job" 10. Click "View batch" (or go to "Staged MARC management" and find the batch you were interested in) 11. Change "Matching rule applied" to "KohaBiblio (999$c)" 12. Change "Action if matching record found" to "Replace existing record with incoming record" 13. Change "Item processing" to "Ignore items"[1] 14. Click "Apply different matching rules" 15. Click "Import this batch into the catalog" 16. Click "View detail of the enqueued job"[2] 17. Note "Number of records updated" is "1" [1] not sure if this is a bug in these patches or if this is pre-existing. It sounds pre-existing iirc... [2] The text in this window isn't very user friendly. I think we could use more job-specific text. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #24 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #15)
To continue I need people to commit to an involvement on these patches. To start I would like someone familiar with the import process to test the workflow and see if they are happy.
I'm happy enough from a user perspective I think. It could probably use a bit more polishing but it looks like the guts are there. -- You are receiving this mail because: You are watching all bug changes.
From a dev perspective, I've been wanting to do clean asynchronous MARC importing as part of OAI-PMH harvesting, but I don't think I'd be able to use
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #25 from David Cook <dcook@prosentient.com.au> --- these background jobs. That's OK though. I think those will need to be customized anyway, and any reusable code will need to be refactored later... I wondered if ./misc/migration_tools/bulkmarcimport.pl would be able to use this, but unfortunately that script seems to have its own approach entirely to importing MARC records. So technically... I think this is good enough too. I reckon let's get this into the codebase, and optimizations can happen later. It gets the job done! -- Note that I only tested with 1 MARC record. QA will probably want to test using a larger number of MARC records. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |black23@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Signed Off --- Comment #26 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Oops -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|m.de.rooy@rijksmuseum.nl |testopia@bugs.koha-communit | |y.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #27 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Still applies. Might have a look rightaway.. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #28 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to David Cook from comment #25)
I reckon let's get this into the codebase, and optimizations can happen later. It gets the job done!
Without referring to this code specifically, but we did that before and what happened? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #29 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to David Cook from comment #25)
Note that I only tested with 1 MARC record. QA will probably want to test using a larger number of MARC records.
Hmm. Lets defer testing to the QA phase? :) I would say: back to NSO, and add another signoff? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #30 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #15)
With those 3 patches I think most of the work is done.
To continue I need people to commit to an involvement on these patches. To start I would like someone familiar with the import process to test the workflow and see if they are happy.
Then we will need to add test coverage and certainly provide some bug fixes.
Who's onboard?
Well, I was. No problem. Seeing no test plan from the author and no unit tests. But a signed off status :) Did you find some experienced user to get user feedback btw ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |nick@bywatersolutions.com --- Comment #31 from Nick Clemens <nick@bywatersolutions.com> --- I tried testing, if I stage with matching it seems to die every time - it is also a bad experience when staging fails - the job remains at 'Started' and the report is present but empty When I import: - The job says 'Started' but also shows 'Completed import of records' - When it finishes, there is no link back to the batch - If it fails it says 'Finished' "Progress 0/0" - From a completed import I can't get back to the background job (maybe not needed?) Lack of error feedback was problematic before, and is worse here because the user is moved to a different section of Koha and the progress is less visible - before you had a stuck bar, now just a not updating screen I like this alot but do think we need to fix these other problems while we do this work - it can be a second bug while we address functionality here, but I would like to see them pushed together in that case -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136216|0 |1 is obsolete| | Attachment #136217|0 |1 is obsolete| | --- Comment #32 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 136338 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136338&action=edit Bug 27421: Use Background job for staging MARC records for import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #33 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 136339 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136339&action=edit Bug 27421: Commit and revert -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |In Discussion --- Comment #34 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Marcel de Rooy from comment #30)
(In reply to Jonathan Druart from comment #15)
With those 3 patches I think most of the work is done.
To continue I need people to commit to an involvement on these patches. To start I would like someone familiar with the import process to test the workflow and see if they are happy.
Then we will need to add test coverage and certainly provide some bug fixes.
Who's onboard?
Well, I was. No problem. Seeing no test plan from the author and no unit tests. But a signed off status :) Did you find some experienced user to get user feedback btw ?
The status should stay 'in discussion', the patches are not ready for master. I will need help for that. (In reply to Nick Clemens from comment #31) Thanks, Nick!
I tried testing, if I stage with matching it seems to die every time - it is also a bad experience when staging fails - the job remains at 'Started' and the report is present but empty
Fixed.
When I import: - The job says 'Started' but also shows 'Completed import of records'
Should be fixed now.
- When it finishes, there is no link back to the batch
Yes, that's one problem I identified. However we don't have the batch id yet. One solution would be to have an ajax script that would get the job's detail and display the link when the job has started. But I would suggest to do it on a separate bug report, if we don't have a better solution. Note that bug 30982 is adding the REST API route for GET /background_jobs and GET /background_jobs/$id
- If it fails it says 'Finished' "Progress 0/0"
Progress bugs should be fixed now.
- From a completed import I can't get back to the background job (maybe not needed?)
Hum, I don't know. If we need it it will be "tricky" (we will need a separate DB column to store that I think).
Lack of error feedback was problematic before, and is worse here because the user is moved to a different section of Koha and the progress is less visible - before you had a stuck bar, now just a not updating screen
Yes, that's why I think we need feedback from regular users. The ajax suggestion above could work but I will be happy if we can come up with a better solution.
I like this alot but do think we need to fix these other problems while we do this work - it can be a second bug while we address functionality here, but I would like to see them pushed together in that case
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30739 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #35 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #34)
Yes, that's why I think we need feedback from regular users.
Let's ask the folk at bug 28128 and bug 31043 since they're having issues with the current MARC import. They can say whether or not this improves things surely. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |victor@tuxayo.net --- Comment #36 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Conflicts, if you don't have time, I can give a try before Monday. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #37 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- I fixed the first conflict Looking at this one: Applying: Bug 27421: Use Background job for staging MARC records for import Applying: Bug 27421: Commit and revert error: sha1 information is lacking or useless (koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/stage_marc_for_import.inc). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #38 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 138642 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=138642&action=edit Bug 27421: Use Background job for staging MARC records for import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136339|0 |1 is obsolete| | --- Comment #39 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 138643 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=138643&action=edit Bug 27421: Commit and revert -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136339|1 |0 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |BLOCKED --- Comment #40 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Manual patching the diffs: patching file Koha/BackgroundJob.pm patching file Koha/BackgroundJob/MARCImportCommitBatch.pm patching file Koha/BackgroundJob/MARCImportRevertBatch.pm patching file debian/templates/apache-shared-intranet-plack.conf patching file koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/marc_import_commit_batch.inc patching file koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/marc_import_revert_batch.inc patching file koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/stage_marc_for_import.inc patching file koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt patching file koha-tmpl/intranet-tmpl/prog/en/modules/tools/manage-marc-import.tt patching file tools/manage-marc-import.pl Hunk #6 NOT MERGED at 275-283. I do not trust the last script. Somehow we ended up touching a considerable lower number of lines.. Will have a look -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #41 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- The diff in the original 35K patch gives me on tools/manage-marc-import.pl: 1 file changed, 47 insertions(+), 22 deletions(-) But there are definitely 210 changes in that patch file. (found with sed) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136338|0 |1 is obsolete| | Attachment #136339|0 |1 is obsolete| | Attachment #138642|0 |1 is obsolete| | Attachment #138643|0 |1 is obsolete| | --- Comment #42 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 138656 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=138656&action=edit Bug 27421: Use Background job for staging MARC records for import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #43 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 138657 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=138657&action=edit Bug 27421: Commit and revert -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #44 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Great it applies now. New module added. What I am missing here completely, is tests ! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #45 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Marcel de Rooy from comment #44)
Great it applies now. New module added. What I am missing here completely, is tests !
How could this be tested easily? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #46 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Jonathan Druart from comment #45)
(In reply to Marcel de Rooy from comment #44)
Great it applies now. New module added. What I am missing here completely, is tests !
How could this be tested easily?
I mean, I am not going to write UI tests, neither a full coverage of the import process. And enqueuing jobs is already tested by t/db_dependent/BackgroundJob.t I could make you happy by making `git grep MARCImportCommitBatch t` returns a couple of occurrences, but I don't know how to have a test that would be useful and that won't require too much time to write. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #47 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- | 9 | finished | 1 | 0 | 51 | marc_import_commit_batch The job size for this job does not get updated? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #48 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 138660 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=138660&action=edit Bug 27421: Fix job's size for import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #49 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 138881 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=138881&action=edit Bug 27421: Add tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #50 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- This patch proves the evidence of bad faith, the tests were pretty trivial to write! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #51 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Please test this one today! QA scheduled for tomorrow :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #52 from Nick Clemens <nick@bywatersolutions.com> --- Stage: - No report number of items found in batch when complete - Staging shows "results" info before completion - in my testing I never saw progress update from "Started" 0/0 until the job finished - On finish: Progress 600532/7655? Import: - Job details always displays: Completed import of records - An import that dies never resolves: Started/Progess 50/437 - Table of results always displays with no info/empty columns - Table is populated when job finished, but is empty on refresh or browse to page Both: - Detailed messages never contains anything - No error feedback - I feel we should directly send the user to the job details: the intermediary screen provides no info and adds a click: The job has been enqueued! It will be processed as soon as possible. View detail of the enqueued job This is so much faster though, and when it works, it works as it did previously I am willing to add provisional sign off to allow QA, but this will need polishing if included in release -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #138656|0 |1 is obsolete| | Attachment #138657|0 |1 is obsolete| | Attachment #138660|0 |1 is obsolete| | Attachment #138881|0 |1 is obsolete| | --- Comment #53 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 139384 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139384&action=edit Bug 27421: Use Background job for staging MARC records for import Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #54 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 139385 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139385&action=edit Bug 27421: Commit and revert Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #55 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 139386 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139386&action=edit Bug 27421: Fix job's size for import Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #56 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 139387 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139387&action=edit Bug 27421: Add tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #57 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 139388 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139388&action=edit Bug 27421: (follow-up) Add exec flag Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #58 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #50)
This patch proves the evidence of bad faith, the tests were pretty trivial to write!
And we can use this comment now to insist that you should write tests whatever claims you make :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |BLOCKED --- Comment #59 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- QAing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #60 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 139448 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139448&action=edit Bug 27421: (QA follow-up) Include manage URL and item counts for import commit Test plan: Run an stage and import. Check import commit job. Click to managed batch. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #61 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 139449 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139449&action=edit Bug 27421: (QA follow-up) Similar changes for revert commit Test plan: Look at job results when reverting imported batch. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #62 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 139450 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139450&action=edit Bug 27421: (QA follow-up) Work in progress: looking at progress/size -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #63 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- QA Status: Thx Nick for your comments. I copied them and added my own. I agree that the status of this patch set is not production ready yet. Since Jonathan cannot finish it, I am willing to coordinate getting this further. But please communicate and help ! I think we should get this in 21.11 and proceed to the other scripts that still need this refactoring to get rid of the ProxyPass stuff. Stage: - No report number of items found in batch when complete => ADDRESSED for Stage and Revert - Staging shows "results" info before completion => STILL LOOKING - in my testing I never saw progress update from "Started" 0/0 until the job finished => STILL LOOKING - On finish: Progress 600532/7655? => STILL LOOKING Import: - Job details always displays: Completed import of records - An import that dies never resolves: Started/Progess 50/437 - Table of results always displays with no info/empty columns - Table is populated when job finished, but is empty on refresh or browse to page Revert: - Finished revert job shows 0 / 0 => PARTIALLY ADDRESSED All: - Detailed messages never contains anything => Room for future extension. No problem for me. - No error feedback => Wasnt this poor already ? - I feel we should directly send the user to the job details: => AGREED. No blocker for me now, but we should either redirect within a few seconds or do it rightaway with a enqueued alert or so. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #64 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Marcel de Rooy from comment #63)
I think we should get this in 21.11 and proceed to the other scripts that still need this refactoring to get rid of the ProxyPass stuff.
Haha 22.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_22_11_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #65 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Nick Clemens from comment #52)
[...]
This is so much faster though, and when it works, it works as it did previously
Out of curiosity about how BackgroundJob works, how does it makes the process faster? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=9354 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #66 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Victor Grousset/tuxayo from comment #65)
(In reply to Nick Clemens from comment #52)
[...]
This is so much faster though, and when it works, it works as it did previously
Out of curiosity about how BackgroundJob works, how does it makes the process faster?
Just an educated guess: using Plack now ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #67 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Marcel de Rooy from comment #66)
(In reply to Victor Grousset/tuxayo from comment #65)
(In reply to Nick Clemens from comment #52)
[...]
This is so much faster though, and when it works, it works as it did previously
Out of curiosity about how BackgroundJob works, how does it makes the process faster?
Just an educated guess: using Plack now ?
And worker still in memory? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #68 from David Cook <dcook@prosentient.com.au> --- I suppose the lack of polling would also make it a faster experience overall. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #69 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Marcel de Rooy from comment #66)
(In reply to Victor Grousset/tuxayo from comment #65)
(In reply to Nick Clemens from comment #52)
[...]
This is so much faster though, and when it works, it works as it did previously
Out of curiosity about how BackgroundJob works, how does it makes the process faster?
Just an educated guess: using Plack now ?
+1, that's what i guessed made it better -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #70 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Thanks all ^^ Oh right, that was one of the few (the only?) things not using Plack. Surprising that this has a big effect on tasks that are long where the time loading Perl and the modules should be a small fraction. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139450|0 |1 is obsolete| | --- Comment #71 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 139797 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139797&action=edit Bug 27421: (QA follow-up) WIP: Adjusting progress and size Might still need a bit of polishing. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #72 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Will try to get this further Monday. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #73 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Hmm. Transaction stuff too in MARCImportCommitBatch.. There is no rollback included as if you would have called txn_do ! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #74 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Marcel de Rooy from comment #73)
Hmm. Transaction stuff too in MARCImportCommitBatch.. There is no rollback included as if you would have called txn_do !
I am wondering if we should still commit after 50 records or so. If the job fails, I do not expect it to commit any records at all now. Especially since it runs in a background worker now. My idea now: parameter that controls intermediate commits ! Any thoughts? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139797|0 |1 is obsolete| | --- Comment #75 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140205 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140205&action=edit Bug 27421: (QA follow-up) Sub finish should respect failed status too Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #76 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140206 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140206&action=edit Bug 27421: (QA follow-up) BatchCommitImportRecords needs param for skipping commits When you submit a background jobs, and it fails, you do not expect partial results in the database. Note that when the Background feature would support a partially completed status, things might change again. Test plan: Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t Note: This serves to verify that it still runs as expected. We will still test the new parameter further on. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140206|0 |1 is obsolete| | --- Comment #77 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140207 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140207&action=edit Bug 27421: (QA follow-up) BatchCommitImportRecords needs param for skipping commits When you submit a background jobs, and it fails, you do not expect partial results in the database. Note that when the Background feature would support a partially completed status, things might change again. Note that the >0 test was superfluous if you check for ^\d+$. Test plan: Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t Note: This serves to verify that it still runs as expected. We will still test the new parameter further on. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #78 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140208 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140208&action=edit Bug 27421: (QA follow-up) Polishing the backgroundjob modules StageMARCForImport: - Rollback in catch - Setting progress, size or status after BatchStageMarcRecords in both try and catch block ImportCommitBatch: - Move setting size back to enqueue moment - Rollback in catch - Setting progress, size or status after BatchStageMarcRecords in both try and catch block ImportRevertBatch: - Move setting size back to enqueue moment - Adding transaction/rollback to module since import routine does not support it. Could be moved later. - Setting progress, size or status after BatchStageMarcRecords in both try and catch block Test plan: Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t Test staging file Bonus: Put a die statement in BatchStageMarcRecords. Test importing batch Bonus: Include some records with an invalid library code; this will trigger an FK exception. Test reverting batch. Bonus: Put a die in BatchRevertRecords. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #79 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Will hopefully finish up tomorrow. Nick, could you have another look later this week? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139384|0 |1 is obsolete| | --- Comment #80 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140223 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140223&action=edit Bug 27421: Use Background job for staging MARC records for import Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139385|0 |1 is obsolete| | --- Comment #81 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140224 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140224&action=edit Bug 27421: Commit and revert Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139386|0 |1 is obsolete| | --- Comment #82 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140225&action=edit Bug 27421: Fix job's size for import Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139387|0 |1 is obsolete| | --- Comment #83 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140226 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140226&action=edit Bug 27421: Add tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139388|0 |1 is obsolete| | --- Comment #84 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140227 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140227&action=edit Bug 27421: (follow-up) Add exec flag Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139448|0 |1 is obsolete| | --- Comment #85 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140228 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140228&action=edit Bug 27421: (QA follow-up) Include manage URL and item counts for import commit Test plan: Run an stage and import. Check import commit job. Click to managed batch. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139449|0 |1 is obsolete| | --- Comment #86 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140229 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140229&action=edit Bug 27421: (QA follow-up) Similar changes for revert commit Test plan: Look at job results when reverting imported batch. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140205|0 |1 is obsolete| | --- Comment #87 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140230 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140230&action=edit Bug 27421: (QA follow-up) Sub finish should respect failed status too Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140207|0 |1 is obsolete| | --- Comment #88 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140231 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140231&action=edit Bug 27421: (QA follow-up) BatchCommitImportRecords needs param for skipping commits When you submit a background jobs, and it fails, you do not expect partial results in the database. Note that when the Background feature would support a partially completed status, things might change again. Note that the >0 test was superfluous if you check for ^\d+$. Test plan: Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t Note: This serves to verify that it still runs as expected. The test plan of the following patch covers the new param. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140208|0 |1 is obsolete| | --- Comment #89 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140232 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140232&action=edit Bug 27421: (QA follow-up) Polishing the backgroundjob modules StageMARCForImport: - Rollback in catch - Setting progress, size or status after BatchStageMarcRecords in both try and catch block ImportCommitBatch: - Move setting size back to enqueue moment - Rollback in catch - Setting progress, size or status after BatchStageMarcRecords in both try and catch block ImportRevertBatch: - Move setting size back to enqueue moment - Adding transaction/rollback to module since import routine does not support it. Could be moved later. - Setting progress, size or status after BatchStageMarcRecords in both try and catch block Test plan: Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t Test staging file Bonus: Put a die statement in BatchStageMarcRecords. Test importing batch Bonus: Include some records with an invalid library code; this will trigger an FK exception. (Reduce the progress from 50 to 1. If your first record would be fine, check if it is NOT imported when the job fails.) Test reverting batch. Bonus: Put a die in BatchRevertRecords. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #90 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140233 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140233&action=edit Bug 27421: (QA follow-up) Template polishing The completed alert needs a condition on status. The record_type variable needs a bit of 'context'. Test plan: Verify if a failed or new job does not have a Completed alert. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #91 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 140234 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140234&action=edit Bug 27421: (QA follow-up) Add redirection to job details Low budget implementation. After 5 seconds we jump to jobs. Small jobs should already be finished. Test plan: Try staging a file. Wait and see if you got redirected. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #92 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- My latest responses to Nick's findings: Stage: - No report number of items found in batch when complete => ADDRESSED for Stage and Revert - Staging shows "results" info before completion => ADDRESSED (at least partially) - in my testing I never saw progress update from "Started" 0/0 until the job finished => OK: TESTED: + = BatchCommitRecords( $import_batch_id, $frameworkcode, 1, + sub { my $job_progress = shift; $self->progress( $job_progress )->store; sleep 10; } ); - On finish: Progress 600532/7655? => ADJUSTED Import: - Job details always displays: Completed import of records => ADDRESSED - An import that dies never resolves: Started/Progess 50/437 => ADDRESSED - Table of results always displays with no info/empty columns => ADDRESSED - Table is populated when job finished, but is empty on refresh or browse to page => PLEASE TEST AGAIN Revert: - Finished revert job shows 0 / 0 => ADDRESSED All: - Detailed messages never contains anything => OK Room for future extension? No problem for me. - No error feedback => OK Wasnt this poor already ? - I feel we should directly send the user to the job details: => ADDRESSED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #93 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- If a patch set needs 8 follow-ups, if it safe to say that it seemed not ready yet as Nick mentioned earlier too ;) Joubu had a good excuse this time. If Nick is available, he is very welcome to have another look. Otherwise Tomas will certainly do. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |BLOCKED --- Comment #94 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- We should probably put this one on top of 31351. Please wait -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Passed QA --- Comment #95 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Marcel de Rooy from comment #94)
We should probably put this one on top of 31351. Please wait
Actually we should be fine already. Applies on top. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |22.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #96 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 22.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |31580 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31580 [Bug 31580] Review Koha::BackgroundJob->finish and 'failed' case workflow -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #97 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 140759 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140759&action=edit Bug 27421: MARC staging/import are long-running tasks Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #98 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Ok, I pushed it to give this a chance (maybe people worry about this change and start giving opinions and patches). Please focus on bug 30982 (which I'd push right away too) so we can come up with a component for displaying progress (once and for all). I agree item information being missing on the run report is bad. Also, I agree we should add a job_id column to the import_batches table so we can link back when the task is complete. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #99 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Tomás Cohen Arazi from comment #98)
Ok, I pushed it to give this a chance (maybe people worry about this change and start giving opinions and patches).
Please focus on bug 30982 (which I'd push right away too) so we can come up with a component for displaying progress (once and for all).
I agree item information being missing on the run report is bad.
Also, I agree we should add a job_id column to the import_batches table so we can link back when the task is complete.
Yes, we live in brave RM times. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |31992 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31992 [Bug 31992] t::lib::Mocks::Zebra still using old stage for import page -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Needs documenting CC| |lucas@bywatersolutions.com --- Comment #100 from Lucas Gass <lucas@bywatersolutions.com> --- This is cool, but no backport for the 22.05.x series -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |31711 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31711 [Bug 31711] When creating order lines "From a new file" you are no longer redirected to acq after import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |31968 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31968 [Bug 31968] Problems with MARC staging/import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|31968 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31968 [Bug 31968] Problems with MARC staging/import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |31891 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31891 [Bug 31891] Regression: show "MARC staging results" with clear link to manage staged bartch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |32780 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32780 [Bug 32780] t/db_dependent/Koha/BackgroundJobs must be moved to t/db_dependent/Koha/BackgroundJob -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33576 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 [Bug 33576] Records are not indexed when imported if using Elasticsearch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com Status|Needs documenting |RESOLVED Resolution|--- |FIXED --- Comment #101 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Already in the manual. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33865 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33865 [Bug 33865] JS error when importing a staged MARC record file -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33972 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33972 [Bug 33972] Possible deadlock in C4::ImportBatch::SetimportBatchStatus from BatchCommitRecords -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Didier Gautheron <didier.gautheron@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33755 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33755 [Bug 33755] Profile used is not saved when importing records -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35156 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35156 [Bug 35156] [Elasticsearch] Fix indexing of batch imported records -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37147 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37147 [Bug 37147] If the MARC import chokes on bad record, it does not fail background job or import -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 --- Comment #102 from David Cook <dcook@prosentient.com.au> --- I'm curious if the code for failed imports was tested, as I'm not sure that I've seen it work... see bug 37147 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org