Re: [Koha-devel] link_bibs_to_authorities.pl can corrupt records
Chris, Many thanks - picking up on you points below: At 09:57 AM 1/7/2012 +1300, Chris Cormack wrote:
I have successfully use that script to link authorities to biblio records after doing a migration. But the way it works is to remove all links and recreate.
Not sure about the "recreate" bit...
Course I didn't do this on a server people were using, which it sounds like you are, to have lost 100 biblios.
"People" are using it. After nearly a year of using Koha 3.2, 3.6.1 is up and running (kudos to the development team, average LAN response time is down from 4.7 secs to well < 1 sec.) I asked for assistance with zebra for authorities, and the assistance from the community allowed me to resolve the difficulties. Then I asked our volunteers to start cataloguing on 3.6.1 to sort out any remaining "update" glitches -- hence the ~100 biblios in the last 24 hours (if you want the nitty-gritty on why we've got to do so many per day, I'll oblige off-list)
Its included for that use case. Jared is working on a much better linking script and he gave you the information how to change it.
Respectfully, it's included in the "current stable release" with no warnings (3.6.1 was "current" when I installed it -- I haven't verified 3.6.2.) and Jared's information came after "horses and barn doors."
Its not a total screwup script, and it didn't demolish the whole db, I think you would find people more willing to help you, in their own free time, in their weekends if you didn't come across so aggressive and accusatory in your tone.
I humbly apologize to anyone who finds me "aggressive and accusatory" -- it's probably the way I was taught to write (concisely, briefly, etc) rather than my feelings. Most people who know me well find me to be a cuddly rather than a grizzly bear. Again my apologies. Best - Paul
Chris On 7 Jan 2012 09:48, "Archives and Collections Society" <<mailto:info@aandc.org>info@aandc.org> wrote: At 03:02 PM 1/6/2012 -0500, Ian Walls wrote:
Paul, The <http://link_bibs_to_authorities.pl>link_bibs_to_authorities.pl script is hardcoded to ALWAYS erase manual links.ÃÂ I'm not sure what the reasoning is behind that.ÃÂ But, every time you run it, any link you've done manually that the script can't automatically figure out will be removed.ÃÂ Nature of the beast (for now, at least).
What is "any link you've done manually"?, please. More than 90% of our input is Z39.50 and "default" item cataloguing.
OK - a total screw-up that is included without any warning whatsoever in the standard distribution of 3.6.1 ??? From the bug report <<http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=5683>http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=5683>
Chris Cormack 2011-06-01 21:13:23 UTC Pushed to master
Jared Camins-Esakov 2011-07-08 14:37:02 UTC Seems to be working now. Closing.
If it's "nature of the beast" why isn't the "beast" mentioned somewhere? What on earth is the reason for this .pl to be included in any distribution?
More...
2012/1/6 Chris Cormack <<mailto:chris@bigballofwax.co.nz>chris@bigballofwax.co.nz> Im assuming you did this on your testing/staging server right? Id restore that from backup, then try applying the patch and see if you get better behaviour. If so, then do it on your production machine.
what "patch"???
And yes -- I can restore from a previous sqldump, and only lose +/- 100 biblios.  We're in pre-production and when we got the zebra indexing working properly, I told our volunteers to start inputting to 3.6.1 *only* (not duplicate on 3.2) so that I could look into any difficulties they experienced as cataloguers.
I had no way at all of knowing that a pl script included in the "latest stable distribution" could possibly demolish the whole db.
Again, what "patch" and where do I find it?
Might I suggest that someone re-opens the bug (I have attempted to annotate it) and removes this dangerous script from public distribution (or at least includes a very visible warning.)
Best - Paul
Chris On 7 Jan 2012 08:47, "Paul" <<mailto:paul.a@aandc.org>paul.a@aandc.org> wrote: Bug 5683 (<http://link_bibs_to_authorities.pl>link_bibs_to_authorities.pl can corrupt records) was signed off some months ago, but is perhaps a little too cryptic for me to follow in detail. à Can someone help? As far as I can see, without remedial action, this is catastrophic. Following difficulties we have had migrating 3.2 to 3.6.1, someone on the users mailing list suggested we should run <http://link_bibs_to_authorities.pl>link_bibs_to_authorities.pl -- (db has ~10k authorities and ~15k biblios) -- so I did just that, and have APPARENTLY DEMOLISHED *EVERY* LINK IN THE DB. à From my notes: paul@nelson:/usr/share/koha$ ./bin/<http://link_bibs_to_authorities.pl>link_bibs_to_authorities.pl Bib authority heading linking report ------------------------------------ Number of bibs checked: ààà14911 Number of bibs modified: ààà 14886 Number of bibs with errors: à0 but - the staff interface no longer correctly finds authorities !!! [1] Manually link (staff client) auth "Lubbock" to bib "Arctic Whalers" -- now linked, 35 other Lubbock bibs not linked. à Is this a question of zebra re-indexing? So: à KOHA_CONF=/etc/koha/koha-conf.xml PERL5LIB=/usr/share/koha/lib ./bin/migration_tools/<http://rebuild_zebra.pl>rebuild_zebra.pl -a -r -v and à KOHA_CONF=/etc/koha/koha-conf.xml PERL5LIB=/usr/share/koha/lib ./bin/migration_tools/<http://rebuild_zebra.pl>rebuild_zebra.pl -b -r -v -x koha@nelson:/usr/share/koha$ ./bin/<http://link_bibs_to_authorities.pl>link_bibs_to_authorities.pl ... processed 100 records / ... / ... processed 14900 records Bib authority heading linking report ------------------------------------ Number of bibs checked: ààà14911 Number of bibs modified: ààà 1 Number of bibs with errors: à0 koha@nelson:/usr/share/koha$ All this did was to *un-link" the bib/auth link that I had manually entered at [1] above. Could someone involved with signing off bug 5683 please explain why link==unlink? Help - please. Paul _______________________________________________ Koha-devel mailing list <mailto:Koha-devel@lists.koha-community.org>Koha-devel@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : <http://www.koha-community.org/>http://www.koha-community.org/ git : <http://git.koha-community.org/>http://git.koha-community.org/ bugs : <http://bugs.koha-community.org/>http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list <mailto:Koha-devel@lists.koha-community.org>Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : <http://www.koha-community.org/>http://www.koha-community.org/ git : <http://git.koha-community.org/>http://git.koha-community.org/ bugs : <http://bugs.koha-community.org/>http://bugs.koha-community.org/
-- Ian Walls Lead Development Specialist ByWater Solutions ALA Midwinter Booth #2048 Phone # (888) 900-8944 <http://bywatersolutions.com>http://bywatersolutions.com <mailto:ian.walls@bywatersolutions.com>ian.walls@bywatersolutions.com Twitter: @sekjal _______________________________________________ Koha-devel mailing list <mailto:Koha-devel@lists.koha-community.org>Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : <http://www.koha-community.org/>http://www.koha-community.org/ git : <http://git.koha-community.org/>http://git.koha-community.org/ bugs : <http://bugs.koha-community.org/>http://bugs.koha-community.org/
--- Archives and Collections (ACS) Society 205, Main Street, Picton, Ontario, K0K 2T0, Canada <http://www.aandc.org/>http://www.AandC.org Canadian Charitable Organization 88721 9921 RR0001 Dedicated to maritime conservation and education.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://UltraMarine.ca>, <http://AandC.org> and <http://MarDoc.ca>
Just what are probably obvious observations here: 1. *Always* run *any* script (stable or otherwise) which works on data, on a test db (read "one into which zero records are being added) *first and before* running it on a live (in *any* sense of the term) db. 2. If my catalogers were doing 100+ bibs per day, I'd be running a mysqldump on the db while they were at lunch and when they quit for the day (and probably anytime they took a coffee break, too). Just my $0.02 worth. Kind Regards, Chris 2012/1/6 Paul <paul.a@aandc.org>:
Chris,
Many thanks - picking up on you points below:
At 09:57 AM 1/7/2012 +1300, Chris Cormack wrote:
I have successfully use that script to link authorities to biblio records after doing a migration. But the way it works is to remove all links and recreate.
Not sure about the "recreate" bit...
Course I didn't do this on a server people were using, which it sounds like you are, to have lost 100 biblios.
"People" are using it. After nearly a year of using Koha 3.2, 3.6.1 is up and running (kudos to the development team, average LAN response time is down from 4.7 secs to well < 1 sec.) I asked for assistance with zebra for authorities, and the assistance from the community allowed me to resolve the difficulties. Then I asked our volunteers to start cataloguing on 3.6.1 to sort out any remaining "update" glitches -- hence the ~100 biblios in the last 24 hours (if you want the nitty-gritty on why we've got to do so many per day, I'll oblige off-list)
Its included for that use case. Jared is working on a much better linking script and he gave you the information how to change it.
Respectfully, it's included in the "current stable release" with no warnings (3.6.1 was "current" when I installed it -- I haven't verified 3.6.2.) and Jared's information came after "horses and barn doors."
Its not a total screwup script, and it didn't demolish the whole db, I think you would find people more willing to help you, in their own free time, in their weekends if you didn't come across so aggressive and accusatory in your tone.
I humbly apologize to anyone who finds me "aggressive and accusatory" -- it's probably the way I was taught to write (concisely, briefly, etc) rather than my feelings. Most people who know me well find me to be a cuddly rather than a grizzly bear. Again my apologies.
Best - Paul
Chris
On 7 Jan 2012 09:48, "Archives and Collections Society" <info@aandc.org> wrote: At 03:02 PM 1/6/2012 -0500, Ian Walls wrote:
Paul, The link_bibs_to_authorities.pl script is hardcoded to ALWAYS erase manual links. I'm not sure what the reasoning is behind that. But, every time you run it, any link you've done manually that the script can't automatically figure out will be removed. Nature of the beast (for now, at least).
What is "any link you've done manually"?, please. More than 90% of our input is Z39.50 and "default" item cataloguing.
OK - a total screw-up that is included without any warning whatsoever in the standard distribution of 3.6.1 ??? From the bug report <http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=5683>
Chris Cormack 2011-06-01 21:13:23 UTC Pushed to master
Jared Camins-Esakov 2011-07-08 14:37:02 UTC Seems to be working now. Closing.
If it's "nature of the beast" why isn't the "beast" mentioned somewhere? What on earth is the reason for this .pl to be included in any distribution?
More...
2012/1/6 Chris Cormack <chris@bigballofwax.co.nz> Im assuming you did this on your testing/staging server right? Id restore that from backup, then try applying the patch and see if you get better behaviour. If so, then do it on your production machine.
what "patch"???
And yes -- I can restore from a previous sqldump, and only lose +/- 100 biblios.  We're in pre-production and when we got the zebra indexing working properly, I told our volunteers to start inputting to 3.6.1 *only* (not duplicate on 3.2) so that I could look into any difficulties they experienced as cataloguers.
I had no way at all of knowing that a pl script included in the "latest stable distribution" could possibly demolish the whole db.
Again, what "patch" and where do I find it?
Might I suggest that someone re-opens the bug (I have attempted to annotate it) and removes this dangerous script from public distribution (or at least includes a very visible warning.)
Best - Paul
Chris On 7 Jan 2012 08:47, "Paul" <paul.a@aandc.org> wrote: Bug 5683 (link_bibs_to_authorities.pl can corrupt records) was signed off some months ago, but is perhaps a little too cryptic for me to follow in detail.  Can someone help? As far as I can see, without remedial action, this is catastrophic. Following difficulties we have had migrating 3.2 to 3.6.1, someone on the users mailing list suggested we should run link_bibs_to_authorities.pl -- (db has ~10k authorities and ~15k biblios) -- so I did just that, and have APPARENTLY DEMOLISHED *EVERY* LINK IN THE DB.  From my notes: paul@nelson:/usr/share/koha$ ./bin/link_bibs_to_authorities.pl Bib authority heading linking report ------------------------------------ Number of bibs checked:    14911 Number of bibs modified:    14886 Number of bibs with errors:  0 but - the staff interface no longer correctly finds authorities !!! [1] Manually link (staff client) auth "Lubbock" to bib "Arctic Whalers" -- now linked, 35 other Lubbock bibs not linked.  Is this a question of zebra re-indexing? So:  KOHA_CONF=/etc/koha/koha-conf.xml PERL5LIB=/usr/share/koha/lib ./bin/migration_tools/rebuild_zebra.pl -a -r -v and  KOHA_CONF=/etc/koha/koha-conf.xml PERL5LIB=/usr/share/koha/lib ./bin/migration_tools/rebuild_zebra.pl -b -r -v -x koha@nelson:/usr/share/koha$ ./bin/link_bibs_to_authorities.pl ... processed 100 records / ... / ... processed 14900 records Bib authority heading linking report ------------------------------------ Number of bibs checked:    14911 Number of bibs modified:    1 Number of bibs with errors:  0 koha@nelson:/usr/share/koha$ All this did was to *un-link" the bib/auth link that I had manually entered at [1] above. Could someone involved with signing off bug 5683 please explain why link==unlink? Help - please. Paul _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Ian Walls Lead Development Specialist ByWater Solutions ALA Midwinter Booth #2048 Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Archives and Collections (ACS) Society 205, Main Street, Picton, Ontario, K0K 2T0, Canada http://www.AandC.org Canadian Charitable Organization 88721 9921 RR0001 Dedicated to maritime conservation and education.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://UltraMarine.ca>, <http://AandC.org> and <http://MarDoc.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Paul (and Koha-devel), 2012/1/6 Paul <paul.a@aandc.org>
I have successfully use that script to link authorities to biblio records after doing a migration. But the way it works is to remove all links and recreate.
Not sure about the "recreate" bit...
Agreed. That's why I'm working on bug 7284. Incidentally (this is for the whole list), I *always* run anything that is supposed to make bulk changes for the first time on a test database (preferred) or using some sort of "test" option. For link_bibs_to_authorities.pl, that option is --test: "only test the authority linking and report the results; do not change the bib records." Is it slow and annoying? Yes. It is better than losing data? Oh my goodness yes. Having worked on a project where a cowboy decided to run a bulk modification script without testing first, I can speak with certainty about that ("Jared, why aren't there any records in the database?" is not something I ever want to hear again). Its included for that use case. Jared is working on a much better linking
script and he gave you the information how to change it.
Respectfully, it's included in the "current stable release" with no warnings (3.6.1 was "current" when I installed it -- I haven't verified 3.6.2.) and Jared's information came after "horses and barn doors."
As it happens, you have encountered this problem at the right time. I have a working replacement linker script that needs testing, if you are so inclined. In pursuit of better results, I would be willing to host a copy of your bibliographic and authority databases so that you can look at the results of the revised linking. Note for the list: THIS OFFER IS OPEN TO ANYONE, first come, first served. I'd like to test over the next week or two, so anyone who's interested, let me know and (if I still have testing slots free) we'll arrange to transfer the files ASAP. Its not a total screwup script, and it didn't demolish the whole db, I
think you would find people more willing to help you, in their own free time, in their weekends if you didn't come across so aggressive and accusatory in your tone.
You may be glad to know that if bug 5683 were still an issue, your system would have been completely unusable, rather than simply unlinked. Regards, Jared -- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
Le 06/01/2012 22:45, Paul a écrit : Hi Paul, it's Paul ;-) I don't understand what you mean here:
(kudos to the development team, average LAN response time is down from 4.7 secs to well < 1 sec.) Do you mean you have 4.7s response time on 3.2 and now have <1sec on 3.6 ? I can't figure why you get a so big boost, I must have misunderstood something ! -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
At 04:07 PM 1/27/2012 +0100, Paul Poulain wrote:
Le 06/01/2012 22:45, Paul a écrit :
Hi Paul, it's Paul ;-)
I don't understand what you mean here:
(kudos to the development team, average LAN response time is down from 4.7 secs to well < 1 sec.) Do you mean you have 4.7s response time on 3.2 and now have <1sec on 3.6 ? I can't figure why you get a so big boost, I must have misunderstood something !
Bonjour cher homonyme, It also involved a change in server, but other usage did not show anywhere near the same gain e.g. a mysql based ap to dynamically list our "used books for sale" from a static webpage went from 0.8 to 0.6 secs. There could be some latency issues here, but I have tried to sort out why Koha improved so much, and while this is not definitive, here are some areas that we noted: - memcached appears to *not* have worked -- or at least not properly -- on 3.2. I think this was a serious issue for zebra - I installed Apache as worker + fastcgi rather than prefork - we use a few custom modules (barcode and call number in particular). For 3.2, we had used them as a subroutine of the standard Koha code -- now I have written them as standalone replacements. Additionally, we have noticed that the "old" 3.2 which we still use as a sandbox and for training purposes, is getting slower and slower. I am *guessing* that it's crud in the mysql which seems to retain a lot of "history" and/or I did not do a clean install after removing wine (a keen volunteer had installed that so as to be able to use MarcEdit which still didn't work), but have not yet had time to look in detail. But the gain is still very real. Our cataloguers' rate has improved dramatically (in part I am sure the psychological effect of a system that appears much slicker.) Avec mes meilleurs voeux pour cette nouvelle année, Paul
Le 27/01/2012 19:41, Paul a écrit :
At 04:07 PM 1/27/2012 +0100, Paul Poulain wrote: - I installed Apache as worker + fastcgi rather than prefork
I'm highly interested by more information for this topic. Could you give me/us some details about your Apache configuration file ? how did you setup fastcgi ? Did it work for both staff & opac ? Thanks for your feedback, it can be a very very important topic in the next weeks. During the hackfest in Marseille, next month, we will have a group dedicated to working on persistency. -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
At 10:59 AM 2/14/2012 +0100, Paul Poulain wrote:
Le 27/01/2012 19:41, Paul a écrit :
At 04:07 PM 1/27/2012 +0100, Paul Poulain wrote: - I installed Apache as worker + fastcgi rather than prefork
I'm highly interested by more information for this topic. Could you give me/us some details about your Apache configuration file ? how did you setup fastcgi ? Did it work for both staff & opac ? Thanks for your feedback, it can be a very very important topic in the next weeks. [snip]
I'll try and dig out my notes, especially on the .conf files. On the install, I just followed the "manual" and had no problems. As to making it work and the various timing advantages (and your question re: OPAC and Staff), I'm in the middle of rebuilding my sand-box and would ask if you could wait a few days on answers as I don't really want to risk upsetting the production server. Best - Paul
Le 16/02/2012 17:49, Paul a écrit :
if you could wait a few days on answers as I don't really want to risk upsetting the production server. Well, my main goal is to have a group working on this topic during the hackfest in Marseille, between march 19th and march 24th, so it can wait a little bit, no worries
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
participants (4)
-
Chris Nighswonger -
Jared Camins-Esakov -
Paul -
Paul Poulain