[Bug 10662] New: Build OAI-PMH Harvesting Client
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Bug ID: 10662 Summary: Build OAI-PMH Harvesting Client Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: new feature Priority: P5 - low Component: Web services Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org Currently, Koha only acts as a OAI-PMH server, I propose to add a harvesting client as well (likely using the HTTP::OAI::Harvester module), so that Koha can ingest records from other data sources (such as digital repositories like Dspace). I've only started reading about it but despite initial reservations about resumption tokens, I think the hardest part will not be with the retrieval of records so much as the parsing of those records into MARC. The Library of Congress does provide some crosswalks (http://www.loc.gov/standards/marcxml/) for converting other metadata formats into MARC21. However, the DC to MARC crosswalk (which is the obvious choice for Dspace) does not produce records of a high quality. So, as part of this new feature, I will also be working on a more complete DC to MARC crosswalk. Their DCMI terms (http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=elements#type) have been mapped reasonably well to the MARC21 LEADER 6th and 7th positions, which improves the quality of the record and helps to produce a "best guess" 008. I'll be looking at adding more datafields and providing better fixed field transformation. Of course, some OAI-PMH repositories serve MARC so this crosswalk might not always be necessary. However, I've looked at Dspace's DC=>MARC crosswalk and it's similar to the LoC one, so I think this will be a valuable addition (both to Koha and to anyone wanting to transform DC to MARC21). As I mentioned, I'm just starting out with OAI-PMH, but I imagine having Koha as a OAI-PMH harvester might be useful in union catalogue situations where other servers might send quality MARC records. -- My plan: 1) Set up a script that is able to continuously harvest records from a OAI-PMH server (likely Dspace or Koha itself for my trials) 2) Set up a database table to handle harvester configuration (such as baseurl, sets, (possibly dates), metadata format, and any pointers to XSLTs. 3) Set up a solid (yet likely basic) DC => MARC XSLT. If anyone has comments or advice, I'd love to hear it. Hopefully, I'll be able to focus on this over the next little while... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Mirko Tietgen <mirko@abunchofthings.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mirko@abunchofthings.net -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Magnus Enger <magnus@enger.priv.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@enger.priv.no -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=6536 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gmcharlt@gmail.com --- Comment #1 from Galen Charlton <gmcharlt@gmail.com> --- (In reply to David Cook from comment #0)
Currently, Koha only acts as a OAI-PMH server, I propose to add a harvesting client as well (likely using the HTTP::OAI::Harvester module), so that Koha can ingest records from other data sources (such as digital repositories like Dspace).
Interesting idea.
I've only started reading about it but despite initial reservations about resumption tokens, I think the hardest part will not be with the retrieval of records so much as the parsing of those records into MARC.
This may be less of a problem in the long run with my plans to allow Koha to support multiple metadata formats (although even once that's available, you may still want the harvester to be able to convert the source metadata into something else). One thing I'd suggest is that the harvester keep a copy of the original metadata record in a database table; that would be more flexible than immediately converting it to MARC and discarding the source data. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- (In reply to Galen Charlton from comment #1)
(In reply to David Cook from comment #0)
Currently, Koha only acts as a OAI-PMH server, I propose to add a harvesting client as well (likely using the HTTP::OAI::Harvester module), so that Koha can ingest records from other data sources (such as digital repositories like Dspace).
Interesting idea.
I'm glad that you approve :). I think you wrote a bit on the subject a few years ago when the OAI-PMH support was first added to Koha, no?
I've only started reading about it but despite initial reservations about resumption tokens, I think the hardest part will not be with the retrieval of records so much as the parsing of those records into MARC.
This may be less of a problem in the long run with my plans to allow Koha to support multiple metadata formats (although even once that's available, you may still want the harvester to be able to convert the source metadata into something else).
I was thinking about that as I started my research, but I'm not sure how far along you are with your plans for metadata diversity. While I'm extremely excited for your work in that area, I suppose I wonder a bit about how feasible (both in terms of time to get there and ultimate function) it is given the current reliance of Koha on MARC data. In any case, like you say, the ability to transform incoming data might still be desired. Either in terms of changing metadata formats or even adding local data to incoming records.
One thing I'd suggest is that the harvester keep a copy of the original metadata record in a database table; that would be more flexible than immediately converting it to MARC and discarding the source data.
Agreed. I was thinking of having a table to keep track of source record identifiers, since I'm still not familiar with resumption tokens, so I could certainly add columns for biblionumber and source metadata record. As you say, that would add flexibility for the future plus give people a source of truth, since data conversion isn't always precise or infallible. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Created attachment 20757 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=20757&action=edit Bug 10662 - Build OAI-PMH Harvesting Client N.B. This feature is still a work in progress. This commit represents my work to date on the OAI-PMH harvester, but it's not complete. While the core classes are operational, I still need to improve the cronjob, the DC to MARC21 XSLT, and the internal workings of the classes (mostly in terms of error handling, edge cases, and reporting). This patch adds several new files to Koha: Overview: 1) C4/OAI/Harvester.pm: This contains 2 classes. The Harvester class sets up the 6 OAI-PMH verbs (2 of which harvest records) and imports records into Koha. The Harvester::Record class is a helper class for processing records, transforming metadata, etc. At the moment, this is hardwired for MARC21 but it's easy enough to expand it out to other flavours (provided there are XSLTs there for the metadata transforms). 2) koha-tmpl/intranet-tmpl/prog/en/xslt/DC2MARC21slim.xsl: This is a XSLT which transforms oai_dc into MARC21. This is a lightweight XSLT based on one I found from the Library of Congress. I improved the Leader and I will endeavour to improve the 008 and map more fields in a more orderly way than this. However, it's a good start. 3) koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim2MARC21enhanced.xsl: This simply strips 999 fields from incoming records, adds the OAI-PMH unique identifier, and adds a 999 field if that unique identifier has already been imported in the past (the 999 is added so that automatic matching and replacement can take place on import). 4) misc/cronjobs/oai_harvester.pl: At the moment, this script takes database config for a OAI-PMH repository to create a Harvester object, then tries each OAI-PMH verb and imports the resulting records into Koha. 5) t/db_dependent/OAI_harvester.t: This is a unit test which uses Koha as the OAI-PMH repository and client in a circular loop. It should test the high level methods and go from retrieving database config to importing records. It uses the "rollback" method so you won't have a bunch of records imported into your database (although your autoincrement will probably go up anyway). Test Plan (For MARC21 users): 0) Apply the patch and run updatedatabase.pl (it will add two new tables to your database: oai_harvest and oai_harvest_repositories). I've documented them in kohastructure.sql. 1) For starters, run the OAI_harvester.t test. It should cycle through all the tests without any problems. If you there are problems, let me know on Bugzilla, in IRC, or via email. 2a) Next, if you feel safe importing records into your database, use the config from the unit test as an example and create an entry in your oai_harvest_repositories table. Be careful with the "import_mode". "Automatic" will automatically stage and import any records harvested via OAI-PMH. "Manual" will stage them but not import them. 2b) Once you're somewhat sure of your config, run oai_harvester.pl. The default is not to use automatic token resumption (so you should only have 50 records in your import batch most likely). You can change this in the cronjob itself. If you fully import this batch, try running "oai_harvester.pl" again. You shouldn't get any results (as you've already imported that batch of records). To try out the "replace/update" feature: update one of the original records you imported (out of that first 50), re-index, and re-run "oai_harvester.pl". You should now get 1 record in your batch which should automatically match the original. In "automatic" mode, it would automatically replace the record (although this can still be reverted in the Manage Staged MARC Records). In "manual" mode, you'll notice in the management interface that there has been a match for that record with the original. 3) To more fully explore: Examine C4/OAI/Harvester.pm. I wrote the POD at the end of August, so it's a bit out of date but it should be mostly accurate. I've tried to include as many comments as possible along the way as well. It's not a very sophisticated module. However, there are a lot of different scenarios that I'm trying to account for. I might've missed some use/edge cases, implemented bad code, or error handling. A place to start might be any FIXME or TODO messages. -- As I said at the top, this is still a work in progress, but I'd be happy to get any feedback or advice on how to proceed with this feature. I'll continue working on it and post updates. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #20757|0 |1 is obsolete| | --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Created attachment 20758 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=20758&action=edit Bug 10662 - Build OAI-PMH Harvesting Client N.B. This feature is still a work in progress. This commit represents my work to date on the OAI-PMH harvester, but it's not complete. While the core classes are operational, I still need to improve the cronjob, the DC to MARC21 XSLT, and the internal workings of the classes (mostly in terms of error handling, edge cases, and reporting). This patch adds several new files to Koha: Overview: 1) C4/OAI/Harvester.pm: This contains 2 classes. The Harvester class sets up the 6 OAI-PMH verbs (2 of which harvest records) and imports records into Koha. The Harvester::Record class is a helper class for processing records, transforming metadata, etc. At the moment, this is hardwired for MARC21 but it's easy enough to expand it out to other flavours (provided there are XSLTs there for the metadata transforms). 2) koha-tmpl/intranet-tmpl/prog/en/xslt/DC2MARC21slim.xsl: This is a XSLT which transforms oai_dc into MARC21. This is a lightweight XSLT based on one I found from the Library of Congress. I improved the Leader and I will endeavour to improve the 008 and map more fields in a more orderly way than this. However, it's a good start. 3) koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim2MARC21enhanced.xsl: This simply strips 999 fields from incoming records, adds the OAI-PMH unique identifier, and adds a 999 field if that unique identifier has already been imported in the past (the 999 is added so that automatic matching and replacement can take place on import). 4) misc/cronjobs/oai_harvester.pl: At the moment, this script takes database config for a OAI-PMH repository to create a Harvester object, then tries each OAI-PMH verb and imports the resulting records into Koha. 5) t/db_dependent/OAI_harvester.t: This is a unit test which uses Koha as the OAI-PMH repository and client in a circular loop. It should test the high level methods and go from retrieving database config to importing records. It uses the "rollback" method so you won't have a bunch of records imported into your database (although your autoincrement will probably go up anyway). Test Plan (For MARC21 users): 0) Apply the patch and run updatedatabase.pl (it will add two new tables to your database: oai_harvest and oai_harvest_repositories). I've documented them in kohastructure.sql. 1) For starters, run the OAI_harvester.t test. It should cycle through all the tests without any problems. If you there are problems, let me know on Bugzilla, in IRC, or via email. 2a) Next, if you feel safe importing records into your database, use the config from the unit test as an example and create an entry in your oai_harvest_repositories table. Be careful with the "import_mode". "Automatic" will automatically stage and import any records harvested via OAI-PMH. "Manual" will stage them but not import them. 2b) Once you're somewhat sure of your config, run oai_harvester.pl. The default is not to use automatic token resumption (so you should only have 50 records in your import batch most likely). You can change this in the cronjob itself. If you fully import this batch, try running "oai_harvester.pl" again. You shouldn't get any results (as you've already imported that batch of records). To try out the "replace/update" feature: update one of the original records you imported (out of that first 50), re-index, and re-run "oai_harvester.pl". You should now get 1 record in your batch which should automatically match the original. In "automatic" mode, it would automatically replace the record (although this can still be reverted in the Manage Staged MARC Records). In "manual" mode, you'll notice in the management interface that there has been a match for that record with the original. 3) To more fully explore: Examine C4/OAI/Harvester.pm. I wrote the POD at the end of August, so it's a bit out of date but it should be mostly accurate. I've tried to include as many comments as possible along the way as well. It's not a very sophisticated module. However, there are a lot of different scenarios that I'm trying to account for. I might've missed some use/edge cases, implemented bad code, or error handling. A place to start might be any FIXME or TODO messages. -- As I said at the top, this is still a work in progress, but I'd be happy to get any feedback or advice on how to proceed with this feature. I'll continue working on it and post updates. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion Patch complexity|--- |Medium patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- Thoughts on things to include: 1) Add a preference/config for using identifier + datestamp OR identifier + datestamp + metadataPrefix as the indicator of the highest order of uniqueness. 2) Add an email feature that tells library staff to check a report which enumerates the status of records imported via OAI-PMH. These can be create (new records), replace (updated records), deleted (for incoming records with a status of deleted), or ambiguous (essentially a new record but linked to multiple existing bib records. These are almost certainly duplicates but require manual merging since it's tough to know which is the real authoritative record). I was thinking perhaps of sending an email containing a link to a Template Toolkit page (so that translation would be possible), which would contain the import/history log. 3) Improving error handling 4) Make the import options more configurable? Although I think the hardcoded options for always replacing a bib match, adding for no match, and ignoring items are probably pretty good. There might be other use cases where people want something different though, so configuration might be a good idea (although ignoring items is fairly essential, as you could duplicate items if you're importing updated records with items). Perhaps the MARC21 XSLT should also strip 952 fields. 5) When using the cronjob, if the "from" date for a repository is "null", check for existing records in Koha, and use the latest "datestamp"? This way we're able to do selective harvesting automatically without having to update our configuration. (I'll probably add this one soon.) 6) Matching rules: a) Check what MARC field that system uses for its biblionumber. Check if there is a matching rule for that field in Koha. If not, create one? Having this matching rule is essential for matching updated records. 7) Improving the DC => MARC conversion (might look at this soon too...it will always be a "best guess" but it has room for a lot of easy improvement) 8) Make an OAI-PMH harvesting web UI. This would allow people to plug in the baseURL for a remote OAI-PMH repository and use the 6 verbs on it. I imagine it being a good way of people getting used to what a OAI-PMH repo has, so that they can set up the automatic cronjob configuration. It could also be a good idea to allow the "ImportRecordsIntoKoha" method for selective harvesting. That said, if it's too easy to use, it might also be abused by someone who doesn't know what they're doing. If there were a GUI, it would need a permission and/or system preference most likely. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Viktor Sarge <viktor.sarge@regionhalland.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |viktor.sarge@regionhalland. | |se --- Comment #6 from Viktor Sarge <viktor.sarge@regionhalland.se> --- I'm excited to see this functionality being developed as we will work with just the scenario you initially mentioned with a union catalogue providing quality records that we import automatically as our book vendor add them to our account in the union catalogue. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #20758|0 |1 is obsolete| | --- Comment #7 from David Cook <dcook@prosentient.com.au> --- Created attachment 21954 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21954&action=edit Bug 10662 - Build OAI-PMH Harvesting Client N.B. This feature is still a work in progress. This commit represents my work to date on the OAI-PMH harvester, but it's not complete... N.B. The test is currently broken (as it was created for an older version of the code). -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Marc Véron <veron@veron.ch> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |veron@veron.ch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 AARON MAY <mpaaron@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |P1 - high CC| |mpaaron@hotmail.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Recently, we've had some renewed interest in this feature, so if I'm able to find time I will be looking at this again. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Koha Team Lyon 3 <koha@univ-lyon3.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |koha@univ-lyon3.fr --- Comment #9 from Koha Team Lyon 3 <koha@univ-lyon3.fr> --- I'm interrested by this feature, if you need someone to test, I'm volunteer. Sonia BOUIS -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21954|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |ASSIGNED --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Koha Team Lyon 3 from comment #9)
I'm interrested by this feature, if you need someone to test, I'm volunteer.
Sonia BOUIS
Great! Thanks, Sonia :). I think that I have the main engine working, but I got distracted by DBIx::Class::Schema and some local work projects, so I haven't finished the UI, the documentation, or the unit tests yet. At the moment, I have another project taking priority over this one, but the OAI-PMH harvester is #2 on my list of projects. Hopefully, I'll be able to look at this soon. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dpavlin@rot13.org -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andreas.hedstrom.mace@sub.s | |u.se --- Comment #11 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- Also very interested in this. Any progress lately? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Andreas Hedström Mace from comment #11)
Also very interested in this. Any progress lately?
Sadly no :( I'm tempted to post my work in progress... but I still have a few things that I need to work out. Here are my current notes to myself: 1) Add a UI in the staff client so that users can configure the repositories from which to harvest (NOTE: this is the change that will create the most merge conflicts, since it touches existing templates) 2) Add the ability to completely re-harvest from a repository (I suppose the thing to do is delete all the records that were derived from that repository, then re-harvest) 3) Improve the whole feature so that the Perl scripts and templates handle the presentation, while most of the logic is driven by the modules. 4) Double-check database structure? 5) Add POD to Harvester.pm and Importer.pm 6) Add unit tests for Harvester.pm and Importer.pm 7) Add necessary changes to updatedatabase.pl 8) Add function to OAI::Importer that will add the item type if it doesn't already exist? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- Actually, I am going to post my code after all. Here's a link to a Github repository that has my current code: https://github.com/minusdavid/Koha/tree/pro_master_oai -- At the moment, I have a test instance of Koha that uses the OAI harvester to pull in all new/changed records from about 6-7 different Koha OAI servers. It seems to work reasonably well. I suppose priorities that I would highlight are: 1) The ability to completely re-harvest from a server. This would involve deleting all the harvested records in Koha, and re-harvesting from the OAI server. 2) Creating a web UI to add/modify/remove repositories for OAI harvesting -- Other than that... it's tidying up the code, adding documentation, and adding unit tests so that this can actually get into Koha. I just don't have time for any of this at the moment. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- For those following along at home, it looks like I'll be starting work on this again soon! My immediate priorities are finishing the web user interface, and adding the ability to re-harvest from an OAI-PMH server. I also need to make some schema changes, which will require some refactoring now, but will save time/energy in the future. Hopefully, in the future we'll be able to harvest authorities and holdings instead of just bibliographic records. After those three things, I'll be posting a revised patch for testing... then probably some follow-up patches with unit tests. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Laura <ljohnson@roseville.ca.us> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ljohnson@roseville.ca.us -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- I think I've nearly arrived at a first draft. On Friday, I'll look at writing out test plans/instructions, and doing some last minute changes. I still need to fix the POD, add unit tests, and add help pages, but I figure that can wait until some initial testing is finished, since the first round of testing might bring more changes to make... I'm excited though! This is probably the closest the OAI-PMH harvester code has ever come to actually being usable for other people than just myself :p -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|Medium patch |Large patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- Created attachment 42446 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42446&action=edit Bug 10662 - Build OAI-PMH Harvesting Client This patch set adds an OAI-PMH harvesting client to Koha. It provides a user interface (UI) for defining external servers from which to harvest records using the OAI-PMH protocol. After it downloads records, it checks the harvest database to see if it needs to add a new record, update an existing record, or delete a record in Koha. _TEST PLAN_ 1) Apply all patches 2) Run updatedatabase.pl (to apply the atomic update) 3) Go to Administration > OAI-PMH servers 4) Click "New OAI-PMH server target" 5) At a minimum, include a valid "Base URL" and a valid "Metadata prefix". 6) Click "Test HTTP and OAI-PMH parameters" 7) If successful, continue with this plan. If unsuccessful, address the warning messages displayed in red before testing the parameters again. 8) At this point, you might want to choose a preferred granularity. All OAI-PMH servers must support YYYY-MM-DD according to the spec, but in practice this isn't always the case, so you may need to choose a more particular granularity (note that this support isn't tested using the "Test" button). 9) You may also want to choose a "From" and "Until" range, at least for the purposes of testing, so that you don't accidentally try downloading thousands or millions of records. (You may also want to download by "Set".) 10) You must set the "Active" parameter to "Active" from "Inactive" for the harvester to work on this server target. 11) Optionally, you may provide a path to a XSLT to transform the incoming data. There is a parameter called "identifier" which is passed to the XSLT engine. This contains the unique OAI-PMH identifier for a record. You may wish to add this to the MARC, especially for the sake of provenance. (You may also want to strip 952, 942, and 999 fields, as well as $9 subfields from incoming records. You may also try the magic "default" keyword here which uses a XSLT I've already written and linked in the backend.) 12) Choose the MARC framework you would like to use (although Default is fine as well). 13) Optionally, you may wish to include a "Original system field". At this time, this has no real purpose. However, in the future, it may be used for linking downloaded holdings records with their original parent record. (e.g. the 004 of a holdings record would link to the 001 of the bibliographic record). This field uses the format of 001 or 999$c with the dollar sign as a subfield delimiter. 14) Click Save 15) You will now see a table containing your entry; click "View". 16) All the numbers on the following screen should be 0. -- 17) Set your environmental variables for KOHA_CONF and PERL5LIB 18) Run "perl /misc/cronjobs/oai/oai_harvester.pl -d -v" to download your records (NOTE: This downloader will run as long as it needs to, so try to only download a few records. Ctrl+C will stop the harvest if it gets out of control.) 19) Revisit the web app as per step #15 20) It should now say "Harvested records waiting to be imported: X" with X being higher than 0. 21) Run "perl /misc/cronjobs/oai/oai_harvester.pl -i -v" to import these records into Koha. 22) The terminal output should indicate the result of the import. This should also be reflected by the webapp as per step #15 (e.g. "Koha records created from harvested records: X"). -- Now, there are a few different scenarios to try: If you control the OAI-PMH repository, try editing a record you've downloaded, and try downloading records again (it might be necessary to change your "From" entry as this should be auto-updated after each harvest), and seeing if your Koha record is updated. If your repository also supports deleted records, try deleting a record that you've already imported into Koha. Koha should get a deletion notice and delete the record from Koha (unless it has items attached). If you delete a record from Koha, but that record still exists upstream, you'll still download updates for that record, but an error will be generated when trying to import into Koha. Each record in this "error" state will be recorded in the "View" section of the UI as "Harvested records in an error state: X". (In the future, I might make it so that the record gets re-added, or add more configuration options to control this behaviour.) If you want to reset the respository harvest (ie delete all your existing harvested records and re-harvest a repository), click "Reset repository harvest" in the "View" screen of the OAI-PMH server target. If errors are encountered while deleting an existing harvest, it should display hyperlinks to the problem records for manual intervention. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #42446|0 |1 is obsolete| | --- Comment #17 from David Cook <dcook@prosentient.com.au> --- Created attachment 42447 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42447&action=edit Bug 10662 - Build OAI-PMH Harvesting Client This patch set adds an OAI-PMH harvesting client to Koha. It provides a user interface (UI) for defining external servers from which to harvest records using the OAI-PMH protocol. After it downloads records, it checks the harvest database to see if it needs to add a new record, update an existing record, or delete a record in Koha. _TEST PLAN_ 1) Apply all patches 2) Run updatedatabase.pl (to apply the atomic update) 3) Go to Administration > OAI-PMH servers 4) Click "New OAI-PMH server target" 5) At a minimum, include a valid "Base URL" and a valid "Metadata prefix". 6) Click "Test HTTP and OAI-PMH parameters" 7) If successful, continue with this plan. If unsuccessful, address the warning messages displayed in red before testing the parameters again. 8) At this point, you might want to choose a preferred granularity. All OAI-PMH servers must support YYYY-MM-DD according to the spec, but in practice this isn't always the case, so you may need to choose a more particular granularity (note that this support isn't tested using the "Test" button). 9) You may also want to choose a "From" and "Until" range, at least for the purposes of testing, so that you don't accidentally try downloading thousands or millions of records. (You may also want to download by "Set".) 10) You must set the "Active" parameter to "Active" from "Inactive" for the harvester to work on this server target. 11) Optionally, you may provide a path to a XSLT to transform the incoming data. There is a parameter called "identifier" which is passed to the XSLT engine. This contains the unique OAI-PMH identifier for a record. You may wish to add this to the MARC, especially for the sake of provenance. (You may also want to strip 952, 942, and 999 fields, as well as $9 subfields from incoming records. You may also try the magic "default" keyword here which uses a XSLT I've already written and linked in the backend.) 12) Choose the MARC framework you would like to use (although Default is fine as well). 13) Optionally, you may wish to include a "Original system field". At this time, this has no real purpose. However, in the future, it may be used for linking downloaded holdings records with their original parent record. (e.g. the 004 of a holdings record would link to the 001 of the bibliographic record). This field uses the format of 001 or 999$c with the dollar sign as a subfield delimiter. 14) Click Save 15) You will now see a table containing your entry; click "View". 16) All the numbers on the following screen should be 0. -- 17) Set your environmental variables for KOHA_CONF and PERL5LIB 18) Run "perl /misc/cronjobs/oai/oai_harvester.pl -d -v" to download your records (NOTE: This downloader will run as long as it needs to, so try to only download a few records. Ctrl+C will stop the harvest if it gets out of control.) 19) Revisit the web app as per step #15 20) It should now say "Harvested records waiting to be imported: X" with X being higher than 0. 21) Run "perl /misc/cronjobs/oai/oai_harvester.pl -i -v" to import these records into Koha. 22) The terminal output should indicate the result of the import. This should also be reflected by the webapp as per step #15 (e.g. "Koha records created from harvested records: X"). -- Now, there are a few different scenarios to try: If you control the OAI-PMH repository, try editing a record you've downloaded, and try downloading records again (it might be necessary to change your "From" entry as this should be auto-updated after each harvest), and seeing if your Koha record is updated. If your repository also supports deleted records, try deleting a record that you've already imported into Koha. Koha should get a deletion notice and delete the record from Koha (unless it has items attached). If you delete a record from Koha, but that record still exists upstream, you'll still download updates for that record, but an error will be generated when trying to import into Koha. Each record in this "error" state will be recorded in the "View" section of the UI as "Harvested records in an error state: X". (In the future, I might make it so that the record gets re-added, or add more configuration options to control this behaviour.) If you want to reset the respository harvest (ie delete all your existing harvested records and re-harvest a repository), click "Reset repository harvest" in the "View" screen of the OAI-PMH server target. If errors are encountered while deleting an existing harvest, it should display hyperlinks to the problem records for manual intervention. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- Created attachment 42448 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42448&action=edit Bug 10662 - DBIx::Class ResultSets for Testing This patch adds the DBIx::Class ResultSets for testing. These are necessary in order for the main OAI-PMH harvester patch to work. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- Hi all! I've finally got something up for testing, so please everyone take some time to test it out. So much has changed since I first started working on this back in 2013, but hopefully it should provide all the functionality that you need. I'm sure that the user interface could use more attention, so I'd love to receive feedback on that. I'd also love to hear back about how the feature works. The key component is the "oai_harvester.pl" cronjob, which will be set up by a system administrator. I don't think there's much that a web user can do to affect that, although I have seen other bugs talking about giving web users control over scheduling tasks. I think web users controlling scheduling would be outside the scope of this bug. Unlike the "Staged MARC Management", there is currently no way of un-importing and re-importing. You can only "reset repository harvest", which will delete all currently harvested records and allow you to schedule a new re-harvest. While I originally was going to leverage the "Staged MARC Management" code, I decided that giving web users control over selectively un-importing and re-importing batches of records harvested via OAI-PMH could be really problematic. That is, you might un-import a batch which deletes 10 records, import a new batch which contains those 10 records, then try to re-import an earlier batch of those 10 records. Even if the (optional) record matching rules were set up perfectly, your Koha records would be wrong; they'd be for an older version of the upstream record. I decided that once a record was added to Koha - all further updates and deletions should be automatic. And if a record was deleted from Koha (other than by "resetting the harvest"), then it could not be re-added; it will instead generate an error (since you can't currently "undelete" a bibliographic record or re-add it with the same biblionumber). However, I'm happy to discuss options for handling records that have been deleted from Koha. There is code that checks if the record has been deleted in Koha, so it would be trivial to add a new record with a new biblionumber, although I'd have to update some other code which expects a unique OAI-PMH identifier to be tied to only 1 Koha biblionumber whereas in this case it would have 2 or more. In fact, I'm happy to discuss every part of this code. Some of you might be interested in improving performance. At the moment, the "oai_harvester.pl" runs synchronously, which means that first all the records need to be downloaded into the database, and then all the records need to be processed and imported into Koha. For initial imports or large imports, this takes hours. However, I've recently gained a lot of experience using POE (Perl Object Environment). Using POE, I could presumably write an asynchronous program which could import records as they're received, rather than waiting for the entire harvest to complete. Unfortunately, POE was removed from Koha's dependencies in the past year or so, but I don't think it would be problematic to add it to the dependencies once again. -- Despite me posting these patches, the work isn't done yet. A keen observer will note that there is a lack of consistency in naming. I sometimes say "oai_client", "oai_server", "oai_target", "oai repository". It's not always exactly clear what I mean, even though I know what I mean. I want to be clear in differentiating this feature from Koha's OAI-PMH server as well. I would be receptive to comments about preferred terminology in both the backend and the web app. Additionally, I also need to do the following: 1) Add unit tests 2) Revise the embedded POD in the code 3) Add help pages (and possibly hints/tips in the templates for web users) I'm going to hold off on these 3 tasks for the moment until we get further into the testing. Otherwise they'll just need to be revised again after more code iterations. (That said, it would have been smart to have written unit tests from the beginning as I built up the code. Alas. Next time.) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P1 - high |P3 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Héctor Eduardo Castro Avalos <hector.hecaxmmx@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=14939 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #42447|0 |1 is obsolete| | --- Comment #20 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 44792 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=44792&action=edit Bug 10662 - Build OAI-PMH Harvesting Client Rebased on master -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #42448|0 |1 is obsolete| | --- Comment #21 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 44793 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=44793&action=edit Bug 10662 - DBIx::Class ResultSets for Testing Rebased on master -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Laurence Lefaucheur <laurence.lefaucheur@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |laurence.lefaucheur@biblibr | |e.com --- Comment #22 from Laurence Lefaucheur <laurence.lefaucheur@biblibre.com> --- I am trying to test OAI harvest on a biblibre Marc21 sandbox following the test plan. I want to import oai_dc records, but the koha records created are always empty (only the leader field is present) I tried with Path to xslt = default, or set empty Record type : biblio Marc Framework : default What kind of metadata is allowed ? I see only this xslt file : MARC21slimFromOAI.xsl Should I previously create a specific oai to marc xslt stylesheet ? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #23 from David Cook <dcook@prosentient.com.au> --- (In reply to Laurence Lefaucheur from comment #22)
I am trying to test OAI harvest on a biblibre Marc21 sandbox following the test plan.
I want to import oai_dc records, but the koha records created are always empty (only the leader field is present)
I tried with Path to xslt = default, or set empty Record type : biblio Marc Framework : default
What kind of metadata is allowed ? I see only this xslt file : MARC21slimFromOAI.xsl Should I previously create a specific oai to marc xslt stylesheet ?
Hi Laurence: Yes, you'll need to create a OAIDC2MARC XSLT in order to transform the metadata into MARC21. Since Koha only supports MARC, the record the OAI-PMH harvester passes to Koha's internal code must also be in MARC. Long ago, I thought about creating a XSLT to convert from oai_dc into MARCXML, but I found it to be a very error-prone process, since oai_dc is such a simple metadata format and MARC is quite complex. I was never happy with the conversion from oai_dc to MARC. With DSpace, I plan to create a DIM to MARC XSLT which I'll probably use on DSpace itself so that I don't have to do anything special on the harvester end. DIM is DSpace's internal metadata format, and I found that to be much better than oai_dc for converting to MARC. So my recommendation is to harvest in MARC, but where that's impossible and you still want access to the record... you'll have to point "Path to XSLT" at a different stylesheet which can convert oai_dc to MARCXML. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #24 from David Cook <dcook@prosentient.com.au> --- I've received a lot of feedback from the National Library of Sweden, so there will probably be more comments going up soon on Bugzilla regarding our discussions about changes to the code that I've posted here. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #25 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- The National Library of Sweden have together with Stockholm University Library provided the funding required for David to finalize his work on the harvester. Stockholm University Library has been testing the OAI-PMH harvester extensively of late, and have provided feedback and been in discussion with Dave about the development of the harvester. Here I’ll try to summarize our discussions. David will probably have to fill in the gaps where needed, and provide further detail! Our use case We are harvesting records from the Swedish union catalogue LIBRIS, which provides records in Marcxml. Today only bibliographic records are harvested, but we hope to add functionality in the future to also allow holdings to be harvested (but this is a separate development and won’t be discussed further here.) We would want to harvest repeatedly and often, preferably every 5 seconds or so, to always have up-to-date records in our local system. Cataloging is done in LIBRIS. Core functionality * The harvester works as intended, where we have tried harvesting record, editing/deleting them at the source and then reharvesting them. All works as intended. * We also tried to delete a record in Koha and then do a harvest – the intended error message is displayed (“Harvested records in error state”). * It’s very good that the HTTP and OAI-PMH parameters for the OAI server target can be tested directly! (I was trying to set up LIBRIS SRU server in Koha the other day and was frustrated that I had to go to cataloging to test whether or not I had set-up correct parameters…) All in all, the harvester works as intended! Major issues Repeated harvests The harvester as built today is made to run one-time harvest or repeating harvest that are long in between – like once every night. For those use cases, performing the scheduling in the GUI and then running the job with the cronjob (the download and the import parameters) is not a problem. But for repeated tasks, this divided responsibility is highly problematic. We would like to have all harvests (or tasks) set from the GUI! To facilitate this, David has proposed to change the functionality of the harvester to work as a daemon instead. The reasons for these is as follows: * Using the daemon, all scheduling can be handled by the GUI * Using the daemon, you could harvest every few seconds. The original intent with the cronjob was that it would be set once and never looked at again. The harvesting would just happen in the background. But since you want more control and to run the harvest every few seconds, a daemon is the way to go. * The key benefit of using the daemon is that you can control it from the GUI and that it can manage the harvests. Trying to set/schedule a cronjob from the GUI would be a bad idea. * If you’re trying to re-harvest every few seconds, a cronjob could easily get out of control. You could easily have competing processes and no way to control them at all. A daemon couldn’t be a communications centre in the way described. The way I envision it, the daemon will communicate with the Web GUI. You could start, stop, and pause harvests. The daemon would also be in charge of the actual harvest, as it could control its own activity. You can’t really control a cronjob. The cron daemon starts cronjobs based on its own unique syntax and that’s it. It’s just a scheduler. It’s not a controller. The daemon I’m talking about would be a controller. You could tell it “STOP 1” and it would stop running the harvest with the 1 identifier. David could preferably provide more detail on the proposed daemon approach. We had some initial reservations about the use of a daemon for the harvester, mainly as this would be a background process that might be hard to evaluate/work with for a systems administrator, to which David replied: * Why would it be hard for a systems administrator to evaluate/work with a daemon? It seems to me that it would actually be easier for sysadmins to evaluate/work with a daemon, as it can be monitored and controlled as a separate process. It’s much easier to control than a cronjob. It would be good to have input from others in the community on the merits of having the harvester run as a daemon! Matching rules At the moment there are not matching rules for the harvester per se. The only matching that is done is based on the OAI-PMH unique identifier. If there’s already a record in Koha with the same title, but not the same OAI-PMH unique identifier, you will get a duplicate. Not having matching rules will essentially make the harvester useless for us, and I would guess anyone harvesting from a union catalogue. We don’t want to add a lot of unnecessary duplicates to our local catalogue. In case of libraries who are already running Koha and would want to start using the harvester, there would be a lot of duplicates (possibly everything!). Also, we do not want to limit libraries to use one source to harvest from – there might be a need in the future to harvest from multiple sources. We suggest that the “Staged Marc Management” tool should be used to actually import the records into Koha – then the matching rules that apply there would be used. Or copying/mirroring this functionality for the harvester. Small issues * Viewing a server target, the page doesn’t have a back button or working breadcrumbs. David has suggested that he might not add a back-button but will fix the breadcrumbs. * The reset repository harvest button should have a warning or a help text next to it, explaining that all harvested records will be removed. * A help text should be added next to the Until parameter, detailing that this should not be set for repeated harvest. Otherwise, as the From parameter is auto-updated with each harvest, Until might be set before From, which will cause to harvester to fail. * More detailed information should be presented under “View”, preferably lists of records imported (where you can click on the bib-id to go to the actual record), lists of deleted records, updated records etc. We will draw up what we would like to see in terms of details and send to David. We can also post it here, if others are interested? * It would be great if multiple sets could be provided for one OAI server. The first time a new server is added, pressing the “Test HTTP and OAI-PMH parameters” will send you back to the OAI-PMH server targets (oai_client.pl) page, like you would expect the save button to do. David has confirmed that this is a bug. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #26 from Viktor Sarge <viktor.sarge@regionhalland.se> ---
Our use case We are harvesting records from the Swedish union catalogue LIBRIS, which provides records in Marcxml. Today only bibliographic records are harvested, but we hope to add functionality in the future to also allow holdings to be harvested (but this is a separate development and won’t be discussed further here.)
We have (as many others) the same use case. Getting holdings would be very great!
All in all, the harvester works as intended!
Great news!
Matching rules At the moment there are not matching rules for the harvester per se. The only matching that is done is based on the OAI-PMH unique identifier. If there’s already a record in Koha with the same title, but not the same OAI-PMH unique identifier, you will get a duplicate.
Not having matching rules will essentially make the harvester useless for us, and I would guess anyone harvesting from a union catalogue. We don’t want to add a lot of unnecessary duplicates to our local catalogue. In case of libraries who are already running Koha and would want to start using the harvester, there would be a lot of duplicates (possibly everything!). Also, we do not want to limit libraries to use one source to harvest from – there might be a need in the future to harvest from multiple sources.
We suggest that the “Staged Marc Management” tool should be used to actually import the records into Koha – then the matching rules that apply there would be used. Or copying/mirroring this functionality for the harvester.
Using the existing import tool sounds like a good plan - then there is a single point to work with for import rules even though we add a new import flow. Much better than building another place to poke around with it's own quirks.
Small issues * Viewing a server target, the page doesn’t have a back button or working breadcrumbs. David has suggested that he might not add a back-button but will fix the breadcrumbs.
Breadcrumbs is good enough if they work correctly and brings you one step up and not two-three steps up in the hierarchy.
* Using the daemon, all scheduling can be handled by the GUI
A GUI is a selling point in my eyes! Everything that lets the library handle their Koha installation by themselves when they don't themselves have the Linux know how is great! Not having to bug the server people about changes is a big plus.
It would be good to have input from others in the community on the merits of having the harvester run as a daemon!
GUI and short intervals for harvesting gets daemon my vote. But that is without a deeper analysis of technical details. Although I know Zebra indexing can now as a daemon which is viewed as a plus so it can't be all that alien a concept. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #27 from David Cook <dcook@prosentient.com.au> --- The first time I started working on this feature, I thought about using “Staged Marc Management”, but there were problems with this which I don't recall 100% (as it was over 2 years ago). I do have some memories though: 1. I wouldn't want the harvests accessible via the "Staged Marc Management" tool, because selective "import"/"undo import" of harvests would be highly problematic. You could import 100 records, unimport 100 records, import 50 records, and then try to re-import those original 100 records which include that 50 record subset. In this case, you might overwrite the newer 50 records with the older 100 records. Of course, you could opt not to overwrite matches... but that relies on there being a good matcher, which there very well might not be. Plus, if you don't overwrite matches and have that setting defined at a OAI-PMH server level, you're never going to get newer records updating older records, which is also bad. 2. The "Staged Marc Management" record matcher relies on Zebra which makes it prone to not always matching correctly. If something hasn't been indexed correctly, you'll get duplicate records. It also relies on that Koha's indexing configuration. In some tests, I've forced the unique OAI-PMH identifier to be placed in the 035$a field... but that field isn't indexed by default. So it would be useless for matching without an update to the Zebra indexing... which can be achieved but it's another point of failure. The matching also relies on import rules defined in Koha. If you have a staff member accidentally delete your OAI-PMH matching rule, you're going to quickly get many many duplicate records. -- I chose to do my own import rules - using only the unique OAI-PMH identifier - because it was the most reliable way of making sure that harvested records weren't duplicated against themselves/each other. In the event that you're harvesting holdings, you also need to have the original bibliographic record in the Koha. That means that if you're having duplicate matching, it must 100% of the time overwrite local bibliographic records. Otherwise, your holdings won't know which bibliographic record to which to bind. If you're using "Staged Marc Management", it's easy to accidentally misconfigure so that you're not overwriting local bibliographic records, and then you have problems again. Another reason I chose to do my own import rules is because I don't think you can trust the user to manage the OAI-PMH harvester configuration completely. -- That all said, I think perhaps the "Staged Marc Management" system might be able to be leveraged... I just don't want it to be configurable by end users, since it needs very particular settings in order to work correctly. Unfortunately, this means that you're going to lose some of the functionality you want, like being able to look at all the records in a harvest. However, the idea of a "harvest" doesn't really make sense if you're using the harvester every few seconds. Each "harvest" might only have 1-2 records in it, so the concept of harvests becomes a bit unhelpful. -- Ultimately, I think we'll need to discuss the import and duplication part of the feature more... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #28 from David Cook <dcook@prosentient.com.au> --- We're also hoping to make the harvester/importer asynchronous which also makes the "harvest" or "batch" concept a bit useless, since it'll be doing lots of individual activities all at the same time. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #29 from David Cook <dcook@prosentient.com.au> --- I've been thinking a bit about the connexion_import_daemon.pl, and how it uses /cgi-bin/koha/svc/import_bib. This uses the "Staged Marc Management" backend without exposing it to users. Unfortunately, I think that service is hard-coded just to work with connexion_import_daemon.pl... so I might need to alter /cgi-bin/koha/svc/import_bib a bit... but that might be an option. I actually quite like that idea overall as it provides a more loosely coupled system. I'm also tempted to change the existing system so that you define OAI-PMH servers, and then create OAI-PMH tasks for those servers. This information would all be stored in the database. Then, when you wanted to run a task, you could click "Run" on the Web UI, and it would send the task to the daemon. I haven't 100% thought out how the Web UI and the daemon will communicate yet. While that above paragraph sounds good, what happens if the daemon dies for some reason? If it requires a message from the Web UI, it'll need a human to restart it. Another thought is to let it access the MySQL database... in that case the Web UI would change a field in the database (like "state" to indicate that it should be running), and then tell the daemon something like "READ 15" to read the task from the database with an ID of 15. That way... if the daemon crashes, a server-side process could detect the crash and then tell the daemon to re-start itself... and when the daemon is re-starting, it could just look in the database for any tasks that it should be running, and get back on track. If it has database access, it's not really that loosely coupled which would be unfortunate... Actually, another idea... the Web UI could send the task, and the daemon could write it away to a temporary file which it cleans up after it's finished a task. If it crashes and gets restarted, it can check its temporary files to see what it was in the middle of doing. Yeah... that's probably a better idea. Another idea would be to use shared memory... but I would need to do some more research into that one. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #30 from David Cook <dcook@prosentient.com.au> --- Thinking more about matching and how complex or even impossible it is. Consider that you have 2 different OAI-PMH servers with 2 matching records and also 1 matching record locally on Koha. Which is the source of truth? You might argue that the harvested records have a higher priority than the local record... so you can overwrite the local record. However, what about the 2 harvested records? Which one takes precedence? What about if they have holdings records? The National Library of Sweden requires that holdings records be partially merged into bibliographic records... that becomes difficult in this scenario. Every time there is a holding record update, you would need to re-create the bibliographic record from the last harvested bibliographic record (otherwise the holding-bibliographic merge would quite quickly get duplicated or otherwise incorrect fields). I suppose you could choose the most recent bibliographic record as the highest priority, and you could blindly merge holdings into that bibliographic record on each update... You'd have to set up a relationship somewhere between the holdings and the bibliographic record though and this gets tough because the holdings from one OAI-PMH server aren't going to map to that bibliographic record using the 004/001 mechanism. That is... Holdings A 004 refers to Bibliographic A 001, so there is a link there. However, Holdings B 004 refers to Bibliographic B 001 which we're discarding as it's a "duplicate". So we need to have a linkage somewhere between Holdings B and Bibliographic A 001 or preferably Bibliographic A 999$c. I think that might be possible, but certainly not with Koha's existing import mechanisms. -- Importing holdings are going to have other issues as well like how to enforce barcode uniqueness... and how to manage values in records that don't exist in Koha. I also need to use my special OAI import system for managing holdings imports, because there will be no reference to the OAI-PMH unique identifier in the Koha item MARXML, so there's no way to use the existing import system to check if that item already exists. I also don't think there's any way to replace an item record using this system unless it shares the same 952$9. I might be wrong, I haven't investigated that issue thoroughly, but I bet I'm right, as it's a tough one. It's also worth reviewing the section marked "Embedded Holdings Information" in http://www.loc.gov/marc/holdings/hd852.html or http://www.loc.gov/marc/bibliographic/bd852.html. Part of the difficulty with the holdings is the fact that Koha doesn't support MARC21 Format for Holdings Data (MFHD). It would be much easier if it did, although there would still be the problems with the source of truth when merging bibliographic records. Merging records and de-duplicating is one thing when your system is relatively static or updated semi-manually, but when you're importing and auto-merging records at a speed of X records every 2 seconds, you're probably going to run into problems. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #31 from David Cook <dcook@prosentient.com.au> --- _HOLDINGS_ Diagramming this now... OAI ID -> Koha ID -> Original 001 -> Parent 001 oai::1 -> bib1 -> 1a -> null oai::2 -> bib1 -> 1b -> null oai::3 -> item 3 -> 3a -> 1a oai::4 -> item 4 -> 4a -> 1b So here we've downloaded oai::1 and added it as bib1. We've downloaded oai::2 and determined that it is a duplicate of bib1. We can either overwrite bib1 or we can simply link to it. We've downloaded oai::3. It's original parent 001 is 1a, so we can link oai::3 to bib1. We've downloaded oai::4. It's original parent is 1b, so we can link oai::4 to the entry for oai::2, which is bib1. In this case, the only problems we have are determining what makes a match, and determining whether oai::1 or oai::2 should provide the metadata for bib1. We probably need another field to say which OAI record is the source of truth for bib1. I might be able to use the existing C4::Matcher() for this... It's worth noting that the downloaded metadata will need to be used every time there's an item update, because the National Library of Sweden requires that item-level data be merged into the host bibliographic record... and the only way to do that cleanly is to start with a virgin bibliographic record each time. (Otherwise, when you change a 863 in a holdings record, you won't be updating a 863 in the bib record, you'll be adding a new one and the old one will stay there incorrectly.) Every holdings record will also need to be merged in as well, which could probably cause load problems with records with lots of holdings... Actually, I'm not sure how that's even possible now that I think about it... since you're harvesting from the holdings endpoint every 2 seconds... I suppose you could queue updates to a bibliographic record from the holdings records... but that wouldn't make sense as every updated holdings record would require a full update of the bibliographic record from all holdings records... so any update would need to be processed. I suppose you could queue updates in terms of... if there's a holdings-bibliographic merge in progress, don't start another one as things will explode... still seems like a potentially intensive operation. -- Problems to consider and solve (anyone can chime in here): 1) Precedence of bibliographic-bibliographic merges 2) Merging holdings records into bibliographic records (e.g. 852 and 863 into the bibliographic record... not 952 into bibliographic record) 3) Any local changes to a record will be erased by future downloaded updates -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #32 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- Just a quick note: the idea of having a harvest running every 2 or 3 seconds is my personal preference. If it is not possible due to potential conflicts or heavy loads, then this could be revised. I would be ok with an harvest once every 10 seconds, every 30 seconds or even once a minute. But preferably not much slower than that... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Leif Andersson <leif.andersson@sub.su.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |leif.andersson@sub.su.se -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #33 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I was told recently that 2-3 seconds is quite standard for OAI-PMH harvests. I think a problem could occur if Zebra is involved in matching as you have to make sure the indexes have caught up before you can reliably match. Say a record is changed at the source twice in a very short timeframe... or added and then changed again, included in 2 harvests... but not yet indexed when the second runs, etc. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #34 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- Yes, if we can avoid using Zebra for matching that would probably be for the best. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #35 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- (In reply to David Cook from comment #29)
I'm also tempted to change the existing system so that you define OAI-PMH servers, and then create OAI-PMH tasks for those servers. This information would all be stored in the database. Then, when you wanted to run a task, you could click "Run" on the Web UI, and it would send the task to the daemon.
To me, this sounds like the way to go! In our case, we would set the task of repeated harvest - and not look at it again unless there are problems. =)
I haven't 100% thought out how the Web UI and the daemon will communicate yet. While that above paragraph sounds good, what happens if the daemon dies for some reason? If it requires a message from the Web UI, it'll need a human to restart it.
Actually, another idea... the Web UI could send the task, and the daemon could write it away to a temporary file which it cleans up after it's finished a task. If it crashes and gets restarted, it can check its temporary files to see what it was in the middle of doing. Yeah... that's probably a better idea.
This too sounds like good approach to me. Some sort of fail-safe to have the harvester pick up and continue what it was doing if it has crashed - but it should be set from the UI to begin with! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #36 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- (In reply to David Cook from comment #30)
Thinking more about matching and how complex or even impossible it is.
Consider that you have 2 different OAI-PMH servers with 2 matching records and also 1 matching record locally on Koha.
Which is the source of truth?
You might argue that the harvested records have a higher priority than the local record... so you can overwrite the local record.
However, what about the 2 harvested records? Which one takes precedence?
I would think that this could be solved by adding a "priority option" when adding new servers. Then the user would decide which source should be ranked higher in terms of conflict/duplicates. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #37 from Viktor Sarge <viktor.sarge@regionhalland.se> ---
You might argue that the harvested records have a higher priority than the local record... so you can overwrite the local record.
However, what about the 2 harvested records? Which one takes precedence?
I would think that this could be solved by adding a "priority option" when adding new servers. Then the user would decide which source should be ranked higher in terms of conflict/duplicates.
Relevant to this discussion might be the fact that we built both tracking of changes to MARC-records with the possibility to roll back changes + a system for setting rules for who can change what fields in MARC records. I'll dig up the relevant threads. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #38 from Viktor Sarge <viktor.sarge@regionhalland.se> --- Write protecting MARC fields based on source of import http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14957 History for MARC records. Roll back changes on a timeline or per field. http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14367 I'm not quite certain if this should be "See also" connections - what say ye? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #39 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think the discussion is very interesting, but I am a bit worried about the handling of a record from multiple sources. Maybe it would be good to agree on a basic version of the harvester for a first implementation and then enhance it for more complicated scenarios step by step? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #40 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #39)
I think the discussion is very interesting, but I am a bit worried about the handling of a record from multiple sources. Maybe it would be good to agree on a basic version of the harvester for a first implementation and then enhance it for more complicated scenarios step by step?
I'm inclined to agree with Katrin. I think it makes more sense to have a solid basic version before we try to over-complicate things. I've actually thought of another problem with the OAI-PMH import... and that's merging bibliographic records. If you were to merge a local record and a OAI-PMH record, and choose for the local record to be the destination record, the OAI-PMH import would be broken for that record... because the merge functionality has no concept of the OAI-PMH harvest. I like the idea of locking records that have been imported via certain mechanisms, although having that locking be effective across the board would require some rigorous checks in place throughout the code. I wish that there were some sort of tracking to show the source of all records (e.g. original, Z39.50, OAI-PMH, REST API, etc) to help out with that. -- The only problem I can see with providing a basic version is that a basic version might not capture all the data that we need for more complicated future scenarios... and it might make it 10 times harder to implement more complicated code in the future as a result... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #41 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #33)
I was told recently that 2-3 seconds is quite standard for OAI-PMH harvests.
I think a problem could occur if Zebra is involved in matching as you have to make sure the indexes have caught up before you can reliably match. Say a record is changed at the source twice in a very short timeframe... or added and then changed again, included in 2 harvests... but not yet indexed when the second runs, etc.
I agree once again with Katrin. I think I've said before (either here or via email) that using Zebra for matching can be very unreliable. Currently, I use the unique OAI-PMH identifiers to handle all harvested records, and that's quite robust, since that identifier should be persistent. However, that obviously doesn't help with matching OAI-PMH harvested records against local records created via other methods. In the short-term, perhaps merging bibliographic records would have to occur manually. Or maybe a deduplication tool could be created to semi-automate that task... although I think that tool would have to prevent any deletion of OAI-PMH harvested records. Actually, this hearkens back to my previous comment. It would be good if each record had a simple way of identifying its origin. So you couldn't delete a record obtained via OAI-PMH unless its parent repository was deleted from Koha or unless you used a OAI-PMH management tool to delete records for that repository. I think providing this "source" or "origin" would need to be done consistently or rather... extensibly. I wouldn't want it to be OAI-PMH specific as that would be short-sighted. At the moment, everything that goes through svc/import_bib uses a webservices import_batch... but that's not very unique. It would be interesting to have unique identifiers for import sources. So you might use the svc/import_bib with the connexion_import_daemon.pl, or with MARCEdit, or your home-grown script, or whatever. It would be interesting to distinguish those separately... and maybe prevent writes/deletions for records that are entered via connexion_import_daemon.pl and home-grown script XYZ, while leaving ones imported via MARCEdit to be managed however since you just exported some original records and re-imported them via MARCEdit after making some changes. One way of doing that would actually be to use developer keys... so a developer would need to get a key from Koha before using the web service and then the Koha sysadmin could handle the interaction between that service and Koha's internals using that key (e.g. if records are imported via Webservice A, prevent Koha users from doing anything with them). I suppose that's a bit tougher to do with OAI-PMH... but not necessarily. When a new OAI-PMH repository is added, the system could generate a key for it, and use that key for handling the permissions for Koha users... I think that element of the discussion would relate a lot to http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14957... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #42 from David Cook <dcook@prosentient.com.au> --- I did some more thinking today... The daemon, which I'm going to call Icarus as it's way easier to say that OAI-PMH harvester daemon, is going to be the focal point. Clients, such as the Koha web staff client or an icarus-client tool, will send "harvesting tasks" to the daemon as JSON messages via a Unix socket. The daemon will process those JSON messages, periodically perform the tasks to download the records, and then hand the records off to a record processor plugin. The plugin can then do whatever. It can hook directly into Koha, it can send the record to a REST API, or whatever. The JSON message is going to have an "extras" object that can contain data for the plugin, such as the URL of an API to send to. The plan at the moment is to leave it up to individuals to add the plugin to @INC when starting up the daemon. That said, I think about adding some sort of default plugin, so that out of the box users can use Icarus (ie the OAI-PMH harvester) without any additional configuration. I think the plugin approach should allow me to build 90% of the infrastructure we need for the harvester while leaving 10% up to libraries to have custom implementations. In theory, this could also help people with different timelines for their desired implementation. That is, I can provide the core functionality, but you might consider developing your own plugins to handle the record. Anyway, I have to run, but that's where my thoughts (and diagrams) have led me. Oh, the daemon will also be based on the POE module. While it was recently removed from Koha's dependencies as it wasn't being used anyway, I imagine it will be trivial to add it back in. We'll have to ask Galen who was the one who removed it, I think, and who will be the package manager going forward... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #43 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- I think your thoughts for Icarus (oh, the name!) are well thought out and promising. Building a core functionality, with a plugin system on top makes sense to me as it will allow for plenty of customization for individual libraries! But adding a default plugin is probably a good idea too... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #44 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- (In reply to David Cook from comment #41)
I agree once again with Katrin. I think I've said before (either here or via email) that using Zebra for matching can be very unreliable.
Currently, I use the unique OAI-PMH identifiers to handle all harvested records, and that's quite robust, since that identifier should be persistent. However, that obviously doesn't help with matching OAI-PMH harvested records against local records created via other methods.
That potential matching rules should not rely on Zebra I think we can all agree on. But only matching via OAI-PMH identifiers I do not think would work for harvests from union catalogs. If I understand it correctly, it would force all libraries who want to start using Koha with OAI-PMH harvests to migrate using OAI-PMH or having a duplicate of all its records (since none of the local records will have OAI-PMH identifiers). In our case that would be about 1,2 million duplicates.
In the short-term, perhaps merging bibliographic records would have to occur manually. Or maybe a deduplication tool could be created to semi-automate that task... although I think that tool would have to prevent any deletion of OAI-PMH harvested records.
Handling 1,2 million duplicates manually, or even semi-automatically will most likely not be possible. Although it might be difficult technically, I still think some sort of matching rules is necessary.
Actually, this hearkens back to my previous comment. It would be good if each record had a simple way of identifying its origin. So you couldn't delete a record obtained via OAI-PMH unless its parent repository was deleted from Koha or unless you used a OAI-PMH management tool to delete records for that repository.
I think providing this "source" or "origin" would need to be done consistently or rather... extensibly. I wouldn't want it to be OAI-PMH specific as that would be short-sighted.
Marking the source/origin of a record sounds like a good idea to me, if it can be easily incorporated.
At the moment, everything that goes through svc/import_bib uses a webservices import_batch... but that's not very unique. It would be interesting to have unique identifiers for import sources. So you might use the svc/import_bib with the connexion_import_daemon.pl, or with MARCEdit, or your home-grown script, or whatever. It would be interesting to distinguish those separately... and maybe prevent writes/deletions for records that are entered via connexion_import_daemon.pl and home-grown script XYZ, while leaving ones imported via MARCEdit to be managed however since you just exported some original records and re-imported them via MARCEdit after making some changes.
I was going to ask how this would tie in with the development of the REST API, but Davids comment #42 explains that. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #45 from Leif Andersson <leif.andersson@sub.su.se> --- Have you ever considered "exporting" some of the marc fields to a separate mysql table? What I am thinking of is those fields that most likely will be used for duplicate detection: 001, 003, 020, 022, 035 We could than do the necessary matching without involving zebra. In most cases I'd imagine. To be useful this would have to be applied on all imports, not only OAI. Such a table could also be used to save the information of the origin of a record - if that is desired. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #46 from David Cook <dcook@prosentient.com.au> --- (In reply to Leif Andersson from comment #45)
Have you ever considered "exporting" some of the marc fields to a separate mysql table? What I am thinking of is those fields that most likely will be used for duplicate detection: 001, 003, 020, 022, 035 We could than do the necessary matching without involving zebra. In most cases I'd imagine.
Yes, I've thought about this a bit. The 020 and 022 can already be found in biblioitems.isbn and biblioitems.issn respectively. Unfortunately, they store multiple values in the same field which is suboptimal in this case although not impossible to use... For a while, I've been thinking that it would be nice to store the 001 somewhere and perhaps the 035. I think the problem with that is we're in a place where we actually want to be moving away from MARC... not entrenching it further. So I don't think we should really add to the biblio or biblioitems tables. Of course, there could be a way around that by making a generic "metadata" table. Something like... metadata.id, metadata.record_id, metadata.scheme, metadata.qualifier, metadata.value. So that would look like: 1, 1, marc21, 001, 123456789 I think that's actually very similar to what they do in DSpace, and I've seen other library systems store their MARC records in a similar way.
To be useful this would have to be applied on all imports, not only OAI.
Well, it would actually need to be applied to _all records_ rather than _all imports_. You'd need that data filled for all records if you were going to match properly.
Such a table could also be used to save the information of the origin of a record - if that is desired.
Actually, that's a good point. We could do something like: metadata.id, metadata.record_id, metadata.scheme, metadata.qualifier, metadata.value. 2, 1, koha, record_origin, oai-pmh Actually, in retrospect, it would be wise to add another field like "metadata.record_type" for biblio, authority, and item. -- I think there are some potential obstacles to this approach though: 1) Ideally, it would be discussed with the Koha community and Release Manager to see if this table could be used by other existing parts of Koha and new features 2) It would need to be added to the existing record Add/Mod/Del functions. This isn't necessarily a huge obstacle... 3) The table would need to be populated initially... for databases with millions of records, this would be a very time-consuming process. Since it would be an intensive process, I think perhaps it would need to be run at the discretion of a system administrator. I think the "touch_all_biblios.pl" script would actually take care of this process, since we'd be updating the Add/Mod/Del functions, so the "metadata" table would be populated by running that script. 4) How to decide which fields should be "exported" into this table? While we could provide configuration for this, configuration changes would require "touch_all_biblios.pl" to be run again for the "metadata" fields to be generated correctly. Perhaps having a backend configuration file would be the best in this case, as the person editing it would also be someone who could re-generate the "metadata" table. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #47 from David Cook <dcook@prosentient.com.au> --- Leif and Andreas: If I understand correctly, your main use case for matching would be to make sure that records previously imported from the union catalogue aren't duplicated when you start using OAI-PMH, yes? In that case, would matching on the 001 be suitable? You could configure your OAI-PMH importer module to look for "metadata.value == incoming record 001" and update the matching record. That would prevent duplication in this situation. I think in that situation it would also be good to add a "metadata.value" for the unique OAI-PMH Identifier... and only update a record if it has no OAI-PMH identifier or if it has a OAI-PMH Identifier that matches the incoming record. That way, you might prevent different OAI-PMH repositories from updating each other's records if you're matching using 020 instead of 001. Since I want the OAI-PMH importer to be plugin based, the rules will be customizable. But if we're not going to use Zebra or some other third-party index, I think Leif's suggestion of a new "exported" metadata table might be necessary. Failing that "metadata" table... there is the "import_biblios", "import_items", and "import_auths" tables which might be re-purposed. "import_biblios" and "import_auths" both have a "control_number" field which I think may be unused... but that won't work in the case that you already have records in Koha which need to expose data for matching. In an effort to move this matching discussion along, I'm going to submit a message to the koha-devel listserv as well to discuss the "metadata" table idea. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #48 from David Cook <dcook@prosentient.com.au> --- Of course, adding a "metadata" table might not be a great idea, because it would be "yet another place" that we store metadata. There's already biblioitems.marcxml (and the soon to be removed biblioitems.marc), Zebra, biblio/biblioitems/items relational fields. I think "metadata" might be a good idea as a long-term shift, but it might be met with resistance in the short-term. That said, if we're going to move away from MARC, surely we need to start transitioning to infrastructure which is independent of MARC. But that's why I think we need community discussion and a decision (not necessarily consensus) on the best way to proceed. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 jdemuth@roseville.ca.us <jdemuth@roseville.ca.us> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jdemuth@roseville.ca.us -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #49 from Leif Andersson <leif.andersson@sub.su.se> --- (In reply to David Cook from comment #47)
Leif and Andreas:
If I understand correctly, your main use case for matching would be to make sure that records previously imported from the union catalogue aren't duplicated when you start using OAI-PMH, yes?
In that case, would matching on the 001 be suitable?
If we are only importing from one source, 001 would be fine. But if we will be using several sources for our imports, then relying on 001 would sooner or later result in a "false matcning" where we end up having one record overwritten by a totally different one. How reliable would it be to add in 003? In MARC21 003 is the alphanumeric "MARC code for the organization whose control number is contained in field 001". I don't know how this fits UNIMARC, though. David suggested in a mail (to Koha devel list) to move field 001 to 035. Or even 001 + 003 to 035 In doing so some refinements could be done to this matching point (e.g. normalization, if 003 is empty then add what we know about the exporting catalog etc) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #50 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #33)
I was told recently that 2-3 seconds is quite standard for OAI-PMH harvests.
Katrin, who said this to you? Andreas was also interested in every 2-3 seconds, but that doesn't seem very feasible to me. Today, I've tried downloading records, and I can download 21 records from a Swedish server in 4-5 seconds. The OAI-PMH harvester utilizes synchronous code for downloading records, so if you have multiple OAI-PMH servers, it will have to download first from Server A, then Server B, then Server C... and then it will start processing records. If each server takes 5 seconds, that's 15 seconds before you even start processing the first record. I think I might be able to find some asynchronous code for downloading records with Perl, but even then it might take 5 seconds or longer just to download records... that's longer than the ideal 2-3 seconds. Plus, the asynchronous code would require me to stop using the HTTP::OAI module and create my own asynchronous version of it... which would take some time and probably be more error-prone due to the speed which I'm trying to develop. I suppose 21 records might be a lot for a harvest running every 2-3 seconds... I just tried the query "verb=ListRecords&metadataPrefix=marcxml&from=2015-12-01T18:01:45Z&until=2015-12-01T18:01:47Z", and my browser downloaded 4 records in 1 second. I suppose it might only take another 1-2 seconds to process those 4 records and import them into Koha. That's just a guess though, as I haven't written the necessary new processing/importing code yet. I suppose if I'm sending HTTP requests asynchronously and if it only takes 1 second to fetch a handful of records, it might be doable in 2-3 seconds... but the more records to fetch from the server, the longer the download is going to take and that blows out the overall time. If 2-3 seconds is just an ideal, it might not matter if it it takes 5-10 seconds. I'm keen for feedback from potential uses of the OAI-PMH server. How much does time/frequency matter to you? This might be a case of premature optimisation. It might be better for me to focus on building a functional system, and then worry about improving the speed later. I did that recently on a different project, and it worked quite well. I focused the majority of my time on meeting the functional requirements, and then spent a few hours tweaking the code to reach massive increases in performance. However, if the system needs to be re-designed to gain those performance increases, then that seems wasteful. -- Another thought I had was to build an "import_oai" API into Koha, and then perhaps write the actual OAI-PMH harvester using a language which works asynchronously by design, like Node.js. Not that I'm excellent with Node.js. I've written code in my spare time which fetches data from a database and asynchronously updates a third-party REST API, but it's certainly not elegant... and requiring Node.js adds a layer of complexity that the Koha community itself would not want to support in any way shape or form I would think. But we could create an import API and then rely on individual libraries to supply their own OAI-PMH harvesters... although for that to work successfully, we would need standards for conversations between harvesters and importers. I'm thinking the "import_oai" or "import?type=oai" API might be a good idea in any case, although I'm not sure how Apache would cope with being hammered by an OAI-PMH harvester sending it multiple XML records every few seconds. Perhaps it's worthwhile having one daemon for downloading records, and another for importing records. Perhaps it's worth writing a forking server to handle incoming records in parallel. -- Honestly though, I would ask that people think further about the frequency of harvests. Is every 2-3 seconds really necessary? Do we really need it to be able to perform that quickly? If so, I'm open to ideas about how to achieve it. I have lots of ideas as outlined above, but more than happy to hear about suggestions, and even happier to be told not to worry about the speed. Unless people think it's a concern, I'm going to continue development with slower synchronous code. I want to make this harvester as modular as possible, so that future upgrades don't require a rewrite of the whole system. Right now, I see the bottleneck being with the downloading of records and passing those records to a processor/importer. The importer, at least for KB, is going to be difficult in terms of the logic involved, but I'm not necessarily that worried about its speed at this point. So I might try to prototype a synchronous downloader as fast as I can and spend more time on the importer and refactoring existing code. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #51 from Leif Andersson <leif.andersson@sub.su.se> --- (In reply to David Cook from comment #50)
Honestly though, I would ask that people think further about the frequency of harvests. Is every 2-3 seconds really necessary? Do we really need it to be able to perform that quickly?
From my point of view it seems a little contrived to use a OAI harvester for
Well, the use case envisioned by Stockholm UL would in practice ideally involve fetching ONE record every 10 minutes or so! The cataloger will be creating/modifying a bib record and a mfhd record in our union catalog. Next, the cataloger will turn to our local catalog, Koha, expecting to find this record already imported. If there is a way for the harvester to decide which ONE record to get...maybe even with some intervention by the cataloger...? So when this ONE record is asked for, we want it to be a quick process getting it from the source and into Koha. Then nightly more massive harvests could be performed to catch up with other modifications to the union catalog. this kind of job - synching Koha with union catalog just to be able to instantly work locally on a record. But it is what has been recommended to us by the union catalog from which we are importing records. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #52 from David Cook <dcook@prosentient.com.au> --- (In reply to Leif Andersson from comment #51)
(In reply to David Cook from comment #50)
Honestly though, I would ask that people think further about the frequency of harvests. Is every 2-3 seconds really necessary? Do we really need it to be able to perform that quickly?
Well, the use case envisioned by Stockholm UL would in practice ideally involve fetching ONE record every 10 minutes or so! The cataloger will be creating/modifying a bib record and a mfhd record in our union catalog. Next, the cataloger will turn to our local catalog, Koha, expecting to find this record already imported. If there is a way for the harvester to decide which ONE record to get...maybe even with some intervention by the cataloger...? So when this ONE record is asked for, we want it to be a quick process getting it from the source and into Koha.
What's Stockholm UL? So you're saying that the union catalogue will only have updates about once every 10 minutes? Or that the cataloguer will only be accessing a record in the union catalogue and Koha once every 10 minutes? I am including a mechanism for fetching "one" record, so long as the user knows the OAI-PMH identifier they're after. They'll be able to add a task for that. Perhaps a future development could be done to provide an interface in the cataloguing module for adding/updating a single record. In place of that interface, they'll be able to add a task in the same area as the other tasks in order to get the one record...
Then nightly more massive harvests could be performed to catch up with other modifications to the union catalog.
Those nightly harvests would certainly be possible with the current design.
From my point of view it seems a little contrived to use a OAI harvester for this kind of job - synching Koha with union catalog just to be able to instantly work locally on a record. But it is what has been recommended to us by the union catalog from which we are importing records.
What do you mean by "work locally on a record"? The union catalog will be the source of truth, so any modifications to a record in Koha would be overwritten by a change in the union catalogue. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #53 from Magnus Enger <magnus@libriotech.no> ---
What's Stockholm UL?
Probably Stockholm University Library. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #54 from Leif Andersson <leif.andersson@sub.su.se> ---
What's Stockholm UL?
Stockholm University Library
So you're saying that the union catalogue will only have updates about once every 10 minutes? Or that the cataloguer will only be accessing a record in the union catalogue and Koha once every 10 minutes?
As a rough estimate the cataloger will access a record every 10 minutes.
What do you mean by "work locally on a record"? The union catalog will be the source of truth, so any modifications to a record in Koha would be overwritten by a change in the union catalogue.
Sorry for being unclear. I ment item data: barcode, item type... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #55 from David Cook <dcook@prosentient.com.au> --- (In reply to Leif Andersson from comment #54)
What's Stockholm UL?
Stockholm University Library
I had a feeling, but I thought you were at the national library, so I was just a bit confused.
So you're saying that the union catalogue will only have updates about once every 10 minutes? Or that the cataloguer will only be accessing a record in the union catalogue and Koha once every 10 minutes?
As a rough estimate the cataloger will access a record every 10 minutes.
In terms of updates, are we aiming at providing very up-to-date records for just cataloguers or both cataloguers and OPAC users?
What do you mean by "work locally on a record"? The union catalog will be the source of truth, so any modifications to a record in Koha would be overwritten by a change in the union catalogue.
Sorry for being unclear. I ment item data: barcode, item type...
Interesting... I've been wondering about that. While I think we can create Koha item records from MFHD records, I'm not sure how we'd update Koha item records from MFHD records, especially if the MFHD records don't have barcodes. The model I'm working with right now is that the local record is replaced by the incoming OAI-PMH record... which would eliminate any local changes (including barcode, item type, etc). Trying to automatically merge an incoming MFHD record with a local Koha item record might work but it could be error prone. Also, without a unique identifier like a barcode, you can't match incoming MFHD records to Koha item records (which weren't originally imported via OAI-PMH). Leif: Can you explain more about how you envision the OAI harvester working with MFHD records and Koha items? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #56 from David Cook <dcook@prosentient.com.au> --- Just realized that I forgot to respond to this comment... (In reply to Leif Andersson from comment #49)
(In reply to David Cook from comment #47)
Leif and Andreas:
If I understand correctly, your main use case for matching would be to make sure that records previously imported from the union catalogue aren't duplicated when you start using OAI-PMH, yes?
In that case, would matching on the 001 be suitable?
If we are only importing from one source, 001 would be fine. But if we will be using several sources for our imports, then relying on 001 would sooner or later result in a "false matcning" where we end up having one record overwritten by a totally different one.
Agreed. I think that would be a very real risk.
How reliable would it be to add in 003? In MARC21 003 is the alphanumeric "MARC code for the organization whose control number is contained in field 001". I don't know how this fits UNIMARC, though.
It should be trivial to merge the 001 and 003 together to form a 035 like "(OCoLC)814782" (http://www.loc.gov/marc/bibliographic/bd035.html). That would certainly help eliminate that risk of "false matching" mentioned above. However, in the LIBRIS data that I've seen, I haven't seen any examples of a 003. I'm not sure how this fits UNIMARC either. Hopefully some of the French people lurking on this bug can provide some insight there.
David suggested in a mail (to Koha devel list) to move field 001 to 035. Or even 001 + 003 to 035 In doing so some refinements could be done to this matching point (e.g. normalization, if 003 is empty then add what we know about the exporting catalog etc)
Hmm, yeah, that would probably be achievable. However, that wouldn't really help too much, because your local Koha records will be missing that 003/additional exporting catalog information. So the matching will still fail. In terms of your local Koha catalogue, you could add a 003 to all records before starting to use the OAI-PMH harvester. Then either update records in LIBRIS to have a 003 as well, OR have the harvester inject a 003 into incoming records which will match the 003 in your local Koha catalogue. That's probably the way to go... Actually, let me think about that for a second... What do your existing Koha records have for 001 and 003? If they've been previously imported from LIBRIS, do they have the LIBRIS 001 in the Koha 001 field? -- In terms of matching, we basically need to make sure that incoming data can map/match to existing data. If Koha records have 001 but no 003, then we have to be able to match using just the 001 from the incoming record. If Koha records have 001 and 003, then we need to match the 001 and 003 of the incoming record against those... Failing that, we need to match a 035 on the incoming record with a 035 on an existing Koha record. I imagine that none of the records have a 035 field. Perhaps it would be worthwhile to create one of those as well as I described above... that would probably be best (at least for bibliographic records and authority records). As I noted in my other comment, matching item records is going to be tricky, as we won't have the 035 mechanism available, unless we cheat a bit and put it in the 952$i (inventory number) or something like that... -- I admit that I'm starting to think a bit about how to add support for MARC holdings into Koha. While long-term we do want to get rid of MARC, I wonder if it could be useful having a "holdings" database in Zebra as well. In terms of library systems, I imagine there will always be a separation of abstract entities and print/digital holdings. Right now, we're really limited using the 952 field for items in Koha... but we do it that way instead of supporting MFHD because there was no other way of searching Zebra using both "bibliographic" and "holdings" data at the same time if that data was in separate records. I suppose that brings me back to the idea that Koha should really have an intermediary extensible metadata format which is indexed. MARC bibliographic and MARC holdings records could be held separately and then processed into a single intermediary record which is indexed and used for search. Display... we could either use the intermediary record or use the internal system numbers to fetch the original MARC metadata for display. Of course, that would require a significant and separate development effort to achieve... Plus... even if we could match a MARC holdings record to a MARC holdings record... each holdings record can have X items specified within it... so you still need a unique identifier at the item-level in order to do full matching. -- An added problem with using OAI-PMH and items is that items can be on loan or otherwise in a "process" which cannot be affected by changes upstream at the OAI-PMH server. Actually, now that I think about it, there is so much data stored in the 952 item record that cannot be overwritten by an upstream change... What is the ideal scenario for harvesting MFHD records via OAI-PMH? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #57 from Viktor Sarge <viktor.sarge@regionhalland.se> ---
It should be trivial to merge the 001 and 003 together to form a 035 like "(OCoLC)814782" (http://www.loc.gov/marc/bibliographic/bd035.html). That would certainly help eliminate that risk of "false matching" mentioned above.
However, in the LIBRIS data that I've seen, I haven't seen any examples of a 003.
Interesting. I've actually perceived it the other way around - that Libris is actually quite good at sticking "SE-LIBR" into 003 of all their records. (A quick example http://libris.kb.se/bib/14862617?vw=full&tab3=marc) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #58 from Magnus Enger <magnus@libriotech.no> --- (In reply to Viktor Sarge from comment #57)
Interesting. I've actually perceived it the other way around - that Libris is actually quite good at sticking "SE-LIBR" into 003 of all their records.
(A quick example http://libris.kb.se/bib/14862617?vw=full&tab3=marc)
A search for se-libr in Hylte gives 7538 hits: http://hylte.bibkat.se/cgi-bin/koha/opac-search.pl?q=se-libr The total number of records is 64397. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #59 from David Cook <dcook@prosentient.com.au> --- (In reply to Viktor Sarge from comment #57)
It should be trivial to merge the 001 and 003 together to form a 035 like "(OCoLC)814782" (http://www.loc.gov/marc/bibliographic/bd035.html). That would certainly help eliminate that risk of "false matching" mentioned above.
However, in the LIBRIS data that I've seen, I haven't seen any examples of a 003.
Interesting. I've actually perceived it the other way around - that Libris is actually quite good at sticking "SE-LIBR" into 003 of all their records.
(A quick example http://libris.kb.se/bib/14862617?vw=full&tab3=marc)
That's very interesting, Viktor! When I look at http://libris.kb.se/bib/219553?vw=full&tab3=marc, I see a 003. However, when I look at http://data.libris.kb.se/bib/oaipmh?verb=GetRecord&metadataPrefix=marcxml&identifier=http://libris.kb.se/resource/bib/219553, I do not see a 003. There are a fair number of other discrepancies between the catalogue and the OAI-PMH server (which is password protected) it seems. I think the OAI-PMH server is still in beta, so perhaps that's the explanation? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #60 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- (In reply to David Cook from comment #52)
(In reply to Leif Andersson from comment #51)
(In reply to David Cook from comment #50)
Honestly though, I would ask that people think further about the frequency of harvests. Is every 2-3 seconds really necessary? Do we really need it to be able to perform that quickly?
Well, the use case envisioned by Stockholm UL would in practice ideally involve fetching ONE record every 10 minutes or so! The cataloger will be creating/modifying a bib record and a mfhd record in our union catalog. Next, the cataloger will turn to our local catalog, Koha, expecting to find this record already imported. If there is a way for the harvester to decide which ONE record to get...maybe even with some intervention by the cataloger...? So when this ONE record is asked for, we want it to be a quick process getting it from the source and into Koha.
To confuse things a little, I will have to contradict my colleague at Stockholm Univ. Library a little buy saying that I don't see why we would want to replicate functionality already offered by LIBRIS (the Swedish union catalogue, for those lurking this thread) - where you can download records individually and then run batch exports at night - rather than creating something better/faster. For me it is preferable to have the catalogue as up-to-date as possible, since LIBRIS will be the "master" (or source of truth as David calls it) for our data. I would rather want the harvester to run every ten seconds or so (or however fast we can get it), to get all updates made to "our" records. The only drawback I can see from such an approach would be an increased load on the servers, which is not a trivial thing of course, but something that should be manageable. (LIBRIS might have a bigger problem if a lot of Swedish libraries start using OAI-PMH harvesting with this approach, but they have themselves recommended this use and will have to handle it accordingly.) Also, I would prefer if the process of harvesting records can be as automated as possible, not involving any extra steps on the catalogers part. We want to make their cataloging easier - not more complex!
So you're saying that the union catalogue will only have updates about once every 10 minutes? Or that the cataloguer will only be accessing a record in the union catalogue and Koha once every 10 minutes?
Records that we handle, i.e. adding/updating either the bibliographic record or the holdings record (or both), is probably only done around every 5-10 min as Leif say. But changes to the bibliographic records, for which we have holdings attached, made by other Swedish libraries is probably much more often. I will try to look at this in the upcoming days, manually harvesting at close intervals (I'm thinking about both trying a 3 second and 10 second approach) to see how many records are downloaded with each harvest.
I am including a mechanism for fetching "one" record, so long as the user knows the OAI-PMH identifier they're after. They'll be able to add a task for that. Perhaps a future development could be done to provide an interface in the cataloguing module for adding/updating a single record. In place of that interface, they'll be able to add a task in the same area as the other tasks in order to get the one record...
Then nightly more massive harvests could be performed to catch up with other modifications to the union catalog.
Those nightly harvests would certainly be possible with the current design.
As I mentioned above, ideally I would want the harvester to run repeatedly, on short intervals. But other libraries whom are interested in using the harvester might have other ideas on which set-up would be best for them. So having the flexibility of running either way (individual harvest+night more massive harvestes or repeated harvest every 10 seconds or so) would be wonderful! I think Davids idea for a plug-in approach, together with the harvest tasks, will work well here! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #61 from David Cook <dcook@prosentient.com.au> --- I've been doing more thinking about OAI-PMH harvesting of holdings records... I looked at some online documentation for how systems like Voyager link holdings records and item records, and while the information I found was a bit spotty, the linkage seems rather tenuous. It looks like the holdings record will store a location and a call number in a 852 field, and then when creating an item, that location and call number will be pulled into the item. It looks like the item is also linked in some manner to the holdings record. So when you're viewing a bibliographic record in the catalogue, you'll also see data from the holdings record merged together with data from the item record. -- So one bibliographic record can be linked to many holdings records which can be linked to many item records. The problem with harvesting holdings records and trying to do something with them in Koha is that Koha doesn't support MARC holdings records. This leads us to want to create item records directly from holdings records, but holdings records aren't item records. While they store item-level data, they themselves aren't item records. Nor do they appear to have any way of specifying how many items are actually held. They have holdings statements for things like journals, but that's free-form plain text. Not really something that can be used programmatically. -- So I don't think you can actually reliably create items from a holdings record, as you don't know how many items to create, nor do you have anything like a barcode to identify those items. If an item record in Koha had a link to a holdings record, (such as in a 952$H subfield), you could use the holdings record to update select subfields in multiple existing Koha item records though. But that's about it. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #62 from David Cook <dcook@prosentient.com.au> --- It's worth mentioning that my above comment regarding holdings is probably only relevant for some systems like Voyager, and are not relevant for all holdings records... Those comments are based on the following pages: http://library.princeton.edu/departments/tsd/katmandu/voyager/holitmono.html http://library.princeton.edu/departments/tsd/katmandu/voyager/relink.html http://library.princeton.edu/departments/tsd/katmandu/voyager/chgloc.html http://www.library.illinois.edu/cam/training/voyagerscript.html However, it does appear that there are fields in holdings records for individual items: 876$a is an "Internal item number" (http://www.loc.gov/marc/holdings/hd876878.html). While I haven't seen this in LIBRIS holdings data, it is a field/subfield which exists. 876$p could then be used for the barcode. Looking at this example: 852 0#$aTxAM$bStacks$hHD9195.A5$iW5 876 ##$aAAH8128-1-1$c$13.75$pA14802137389 852$a = Location (ie Library) 852$b = Sublocation/Collection (Branch) 852$h = Classification part (start of call number) 852$i = Item part (end of call number) 876$a = Internal item number (itemnumber) 876$c = Cost (cost) 876$p = Piece designation (barcode) However... the 876 subfields are really only useful if they're present. If your record just has a 852 without any actual item-level data... you can't very well do add/match/update items in Koha without it. Even in the event though that you have 876 data, there are hurdles as outlined below. It's not too bad when there is one 852 and one 876 for a single part item: "One copy, one 852 field - Item fields do not need a linking subfield as all apply to the one holdings item. (Holdings are recorded at level 1 or 2 in Leader/17 (Encoding Level).) [See example under subfield $p below]" (http://www.loc.gov/marc/holdings/hd876878.html). However, "Multiple copies, one or more 852 fields - Subfield $3 is used to link item fields to the appropriate copy in 852 field(s). (Holdings are recorded at level 1 or 2 in Leader/17 (Encoding Level).) [See example under subfield $e below]" (http://www.loc.gov/marc/holdings/hd876878.html). But the $3 subfield isn't a coded field. It's free text, which foils any consistent automated linking. It gets even more complicated for "Multiple physical part holdings item": "Holdings in 866-868 (Textual Holdings) fields - Subfield $3 is used to link item fields to the appropriate part specified in fields 866-868. (Holdings are recorded at level 3 or 4 in Leader/17 (Encoding Level).) [See example under subfield $h below] Holdings in 863-865 (Enumeration and Chronology) fields - Subfield $8 is used to link item fields to the appropriate part specified in fields 863-865. Each part (volume or volumes) for which item level information is included requires a separate 863-865 field. (Holdings are recorded at level 3 or 4 with piece designation in Leader/17 (Encoding Level).) [See example under subfield $l below]" -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #63 from David Cook <dcook@prosentient.com.au> --- Finally, I don't know how much sense it makes harvesting MARC holdings records via OAI-PMH. I can understand harvesting MARC authority records and MARC bibliographic records, as both are theoretically very universal and interchangeable. You might need to change the system numbers, but that's about it. However, with holdings records, you're describing something that is local to your particular library. In theory, wouldn't you want to be uploading holdings records to a union catalogue, rather than downloading holdings records from it? I believe that's how it works with the National Library of Australia. We do some automated exports for the National Library of Australia union catalogue. I think some bibliographic details are included for matching purposes, and then we export all the item data so that they can update the holdings in the union catalogue on a regular (I think it's nightly) basis. -- In the case of LIBRIS, it appears minimal information about "items" is included in a holdings record. Really just the sublocation/collection and the call number. I haven't seen any indication as to the number of items, their barcodes, or really anything specific about "items". So I think it would be difficult if not impossible to use the data I've seen to add/update items in Koha. That said, the data in 866 does certainly seem valuable regarding holdings, and I can see the utility in adding that to the local catalogue. Part of me wants to add support for MARC holdings records in Koha, although I imagine there would be resistance to that, as it further entrenches our use of MARC. The problem with adding OAI-PMH support for holdings is that merging holdings data in to bibliographic records would be tricky. Easy to add it initially, but difficult - if not impossible - to reliably update later. But it would be trivially to import the holdings records into a table and link them to an existing bibliographic record. We could then embed holdings records into bibliographic records at index time (like we already do with item records), so that data from holdings records would be searchable and displayable on the detail page and search results page. That would make it rather easy to "merge" MARC holdings records into MARC bibliographic records for search/retrieval and display. It wouldn't help with "items" but at least the holdings data would be in Koha. We could also consider linking items and holdings records... although I think we'd want to think about the long-term implications of that in a non-MARC environment. It probably would be OK... because "more_subfields_xml" really is filling that gap at the moment, and "items" could just be used for more transactional data while additional data could be offloaded to a XML record (like we do for authorities and bibliographic records... and how we probably would with RDF records anyway). Mind you, it would be more complex with holdings, because holdings records can still refer to multiple items... the idea of linking "items" and "holdings" still seems like it's very specialized and ILS-specific which is no good at all... -- Anyway, hopefully these 3 epic-length comments help further the conversation around harvesting MARC holdings records via OAI-PMH. I think it's a very difficult thing... and provides us with challenges not necessarily in terms of "logic" but rather with "data", links, and relationships. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org