Importing items with "foreign" branches fails badly
Hello! When importing a marc file where some items homebranch are not matching a branchcode in db, the import fails. Freeze-fail. Before, it used to just skip those items, but it seems (since 23.05) to just fail silently (with a constraint failure in the log, but nothing for the user). Is it our install or it's a known issue? When running "show engin innodb status", I get some row lock. Looking at the bug list, I found nothing. There are many ways to fix this, and we don't want to head into one to be told "ooh, this should be fixed there instead". * Upstream https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21272 Maybe there could be an option in staging to overwrite the homebranches that do not correspond? * Just ignore the item, as before. But at least leave some info about it in the result page. * Fail as soon as it happens. Do not import and tell the user to correct the data ? The first option doesn't solve bulkmarcimport.pl :( Feedbacks would be appreciated. -- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
I was looking at the Patron import yesterday, and liked that I could specify the categorycode and branchcode for an import batch. Maybe something like that would work for staging and bulkmarcimport.pl, where there is a (command line) option to override certain fields while processing records. On Fri, Aug 25, 2023 at 8:09 AM Philippe Blouin <philippe.blouin@inlibro.com> wrote:
Hello!
When importing a marc file where some items homebranch are not matching a branchcode in db, the import fails. Freeze-fail.
Before, it used to just skip those items, but it seems (since 23.05) to just fail silently (with a constraint failure in the log, but nothing for the user). Is it our install or it's a known issue? When running "show engin innodb status", I get some row lock.
Looking at the bug list, I found nothing. There are many ways to fix this, and we don't want to head into one to be told "ooh, this should be fixed there instead".
- Upstream https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21272 Maybe there could be an option in staging to overwrite the homebranches that do not correspond? - Just ignore the item, as before. But at least leave some info about it in the result page. - Fail as soon as it happens. Do not import and tell the user to correct the data ?
The first option doesn't solve bulkmarcimport.pl :(
Feedbacks would be appreciated. -- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Michael Hafen Washington County School District Technology Department Systems Analyst
Hi, we had this pop up as a problem on IRC in the last few days too, the silent fail/missing feedback is quite annoying. I am not sure if a default would always fix the issue as you might have files with items from different libraries to import. I think a good problem report would be the most helpful. Not only listing the number of items not imported, but also the errors, especially missing or wrong branchcodes with the wrong branchcodes listed. That would help with debugging the import files immensely. As we can undo an incomplete import, it would probably be ok to skip the items instead of failing. I have no strong opinion on skip vs. fail immediately. Hope this helps, Katrin On 25.08.23 16:09, Philippe Blouin wrote:
Hello!
When importing a marc file where some items homebranch are not matching a branchcode in db, the import fails. Freeze-fail.
Before, it used to just skip those items, but it seems (since 23.05) to just fail silently (with a constraint failure in the log, but nothing for the user). Is it our install or it's a known issue? When running "show engin innodb status", I get some row lock.
Looking at the bug list, I found nothing. There are many ways to fix this, and we don't want to head into one to be told "ooh, this should be fixed there instead".
* Upstream https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21272 Maybe there could be an option in staging to overwrite the homebranches that do not correspond? * Just ignore the item, as before. But at least leave some info about it in the result page. * Fail as soon as it happens. Do not import and tell the user to correct the data ?
The first option doesn't solve bulkmarcimport.pl :(
Feedbacks would be appreciated.
-- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website :https://www.koha-community.org/ git :https://git.koha-community.org/ bugs :https://bugs.koha-community.org/
I also wonder where was this new behavior introduced and if it was ever discussed. A system should not change "just because", without prior if not consent, at least debate. Manos Στις Σάβ 26 Αυγ 2023 στις 11:13 πμ ο χρήστης Katrin Fischer < katrin.fischer.83@web.de> έγραψε:
Hi,
we had this pop up as a problem on IRC in the last few days too, the silent fail/missing feedback is quite annoying.
I am not sure if a default would always fix the issue as you might have files with items from different libraries to import.
I think a good problem report would be the most helpful. Not only listing the number of items not imported, but also the errors, especially missing or wrong branchcodes with the wrong branchcodes listed. That would help with debugging the import files immensely.
As we can undo an incomplete import, it would probably be ok to skip the items instead of failing. I have no strong opinion on skip vs. fail immediately.
Hope this helps,
Katrin On 25.08.23 16:09, Philippe Blouin wrote:
Hello!
When importing a marc file where some items homebranch are not matching a branchcode in db, the import fails. Freeze-fail.
Before, it used to just skip those items, but it seems (since 23.05) to just fail silently (with a constraint failure in the log, but nothing for the user). Is it our install or it's a known issue? When running "show engin innodb status", I get some row lock.
Looking at the bug list, I found nothing. There are many ways to fix this, and we don't want to head into one to be told "ooh, this should be fixed there instead".
- Upstream https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21272 Maybe there could be an option in staging to overwrite the homebranches that do not correspond? - Just ignore the item, as before. But at least leave some info about it in the result page. - Fail as soon as it happens. Do not import and tell the user to correct the data ?
The first option doesn't solve bulkmarcimport.pl :(
Feedbacks would be appreciated. -- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com
_______________________________________________ Koha-devel mailing listKoha-devel@lists.koha-community.orghttps://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
I believe this is more likely to be an unintended side affect of some other work done to improve the perfomance and reliability of the import scripts. There was no bug reported as of yet, I suggest to do this as a next step. Katrin On 26.08.23 10:20, Manos PETRIDIS wrote:
I also wonder where was this new behavior introduced and if it was ever discussed. A system should not change "just because", without prior if not consent, at least debate.
Manos
Στις Σάβ 26 Αυγ 2023 στις 11:13 πμ ο χρήστης Katrin Fischer <katrin.fischer.83@web.de> έγραψε:
Hi,
we had this pop up as a problem on IRC in the last few days too, the silent fail/missing feedback is quite annoying.
I am not sure if a default would always fix the issue as you might have files with items from different libraries to import.
I think a good problem report would be the most helpful. Not only listing the number of items not imported, but also the errors, especially missing or wrong branchcodes with the wrong branchcodes listed. That would help with debugging the import files immensely.
As we can undo an incomplete import, it would probably be ok to skip the items instead of failing. I have no strong opinion on skip vs. fail immediately.
Hope this helps,
Katrin
On 25.08.23 16:09, Philippe Blouin wrote:
Hello!
When importing a marc file where some items homebranch are not matching a branchcode in db, the import fails. Freeze-fail.
Before, it used to just skip those items, but it seems (since 23.05) to just fail silently (with a constraint failure in the log, but nothing for the user). Is it our install or it's a known issue? When running "show engin innodb status", I get some row lock.
Looking at the bug list, I found nothing. There are many ways to fix this, and we don't want to head into one to be told "ooh, this should be fixed there instead".
* Upstream https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21272 Maybe there could be an option in staging to overwrite the homebranches that do not correspond? * Just ignore the item, as before. But at least leave some info about it in the result page. * Fail as soon as it happens. Do not import and tell the user to correct the data ?
The first option doesn't solve bulkmarcimport.pl <http://bulkmarcimport.pl> :(
Feedbacks would be appreciated.
-- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website :https://www.koha-community.org/ git :https://git.koha-community.org/ bugs :https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Hi Cate, Hammat did something with bz21272, if someone wants to check it. We're already using it, since the alternative (crash, no feedback) is really too problematic. Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2023-08-26 04:32, Katrin Fischer wrote:
I believe this is more likely to be an unintended side affect of some other work done to improve the perfomance and reliability of the import scripts. There was no bug reported as of yet, I suggest to do this as a next step.
Katrin
On 26.08.23 10:20, Manos PETRIDIS wrote:
I also wonder where was this new behavior introduced and if it was ever discussed. A system should not change "just because", without prior if not consent, at least debate.
Manos
Στις Σάβ 26 Αυγ 2023 στις 11:13 πμ ο χρήστης Katrin Fischer <katrin.fischer.83@web.de> έγραψε:
Hi,
we had this pop up as a problem on IRC in the last few days too, the silent fail/missing feedback is quite annoying.
I am not sure if a default would always fix the issue as you might have files with items from different libraries to import.
I think a good problem report would be the most helpful. Not only listing the number of items not imported, but also the errors, especially missing or wrong branchcodes with the wrong branchcodes listed. That would help with debugging the import files immensely.
As we can undo an incomplete import, it would probably be ok to skip the items instead of failing. I have no strong opinion on skip vs. fail immediately.
Hope this helps,
Katrin
On 25.08.23 16:09, Philippe Blouin wrote:
Hello!
When importing a marc file where some items homebranch are not matching a branchcode in db, the import fails. Freeze-fail.
Before, it used to just skip those items, but it seems (since 23.05) to just fail silently (with a constraint failure in the log, but nothing for the user). Is it our install or it's a known issue? When running "show engin innodb status", I get some row lock.
Looking at the bug list, I found nothing. There are many ways to fix this, and we don't want to head into one to be told "ooh, this should be fixed there instead".
* Upstream https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21272 Maybe there could be an option in staging to overwrite the homebranches that do not correspond? * Just ignore the item, as before. But at least leave some info about it in the result page. * Fail as soon as it happens. Do not import and tell the user to correct the data ?
The first option doesn't solve bulkmarcimport.pl <http://bulkmarcimport.pl> :(
Feedbacks would be appreciated.
-- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website :https://www.koha-community.org/ git :https://git.koha-community.org/ bugs :https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website :https://www.koha-community.org/ git :https://git.koha-community.org/ bugs :https://bugs.koha-community.org/
participants (4)
-
Katrin Fischer -
Manos PETRIDIS -
Michael Hafen -
Philippe Blouin