[Koha-devel] Hide records on Leader 05 = d in OPAC

[David Cook]

Hi Galen:

I think adding a configuration option makes sense in that case. The thought occurred to me at the time, but I figured it would be ubiquitously welcome without one. A configuration option is easy enough to add.

Adding a mechanism for specifying a set of query additions could be interesting, although it would involve some refactoring and re-testing of existing Koha code, which I doubt anyone would want to do.

> I do not find this a compelling use case as stated.  If the goal is to allow
> harvesting and overlay records from an OAI-PMH provider to also delete bibs
> from a Koha database... coding so that the records are
> *actually* deleted seems more direct.

Originally, I thought the same thing, Galen. I thought that I would download the OAI-PMH data, see the deleted status, and then just delete the record from Koha. However, I soon realized that I actually didn't have any way of doing that. The OAI-PMH data only contains the deleted status and the unique OAI-PMH identifier. The update is coming from upstream and it has no knowledge of downstream. So I first need to match that unique OAI-PMH identifier with a record in Koha before I can directly delete the record.

At this point, I think the consensus amongst community members is to use Zebra, the record matching rules, and the MARC import_bib API for ingesting MARC records programmatically. And that works really well for additions and updates. However, it doesn't work so well for deletions. In theory, we could try to make it delete a record as soon as it sees a Leader/05=d, but that runs into the risks you and Colin described where librarians might not realize that Leader/05=d is set. I suppose another option could be added that makes this more explicit, but then we're changing an API which has been in steady use for years.

I suppose I could keep a table tying the OAI-PMH identifier to a Koha biblionumber, but that's error-prone (especially due to bibliographic record merges, local deletions, etc) and it's a lot of data to keep for the single purpose of deleting records.

Additionally, processing the deletions directly is inherently problematic. If there are items, the deletion will fail. I think there might be a few other foreign relationships which will cause bibliographic record deletes to fail. At that point, the third-party is at an impasse. They might know that they have failed to delete a record in Koha, but no one in Koha knows that a record wasn't successfully deleted.

Going back to the added table idea, I suppose I could store the OAI-PMH identifier, and note an error if it fails to delete, and then Koha users can check that. However, that adds more overhead to the import of MARC records via an OAI-PMH provider. I'd have to make another web service to update the OAI-PMH table before trying to send it to import_bib. That's double the HTTP requests though, so maybe I'd just make import_oai and skip import_bib all together, but then I'm duplicating most of the code in import_bib just to support deletions. That seems inefficient and error-prone as well.

I'm reflecting on the wording in your last message, Galen. I suppose the main goal is to harvest records from an OAI-PMH server, and then to add/update MARC records in Koha. That works well. But a secondary goal is to delete records from Koha where they've been deleted upstream in the OAI-PMH server, yes. As I noted above though, upstream doesn't have knowledge of downstream. I suppose I could query Zebra myself using Z39.50 or SRU (although SRU is disabled by default I believe), but then I lose the record matching rules, which are vital to matching MARC record fields with Zebra indexes. But let's say I query Zebra directly, I could then get the biblionumber I need and then use "/svc/bib" to attempt a deletion. Although that still runs into the same problem as before... the automatic deletion might fail... and no one in Koha will know. At that point, I suppose I could try sending an email, or have some other error reporting mechanism to Koha users... but then the added code is simply to explain that a deletion has failed. 

It seems simpler and less error-prone to mark a record as deleted, hide it from the OPAC, and then let Koha users either use searches or reports to find Leader/05=d records to deal with according to their own discretion. I suppose the reports would be difficult on larger databases, but a search would work well. "Record-status=d" would show all records listed as deleted with reasonable efficiency.

I like the idea of just deleting records directly, but I think it's more complex than it appears at first glance. It's not just an issue with OAI-PMH either really. It's an issue any time you try to delete a record without providing feedback to an end-user. 

I am open to suggestions though. 

David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St, Ultimo, NSW 2007

> -----Original Message-----
> From: Galen Charlton [mailto:gmc at esilibrary.com]
> Sent: Wednesday, 13 January 2016 12:48 AM
> To: David Cook <dcook at prosentient.com.au>
> Cc: Koha Devel <koha-devel at lists.koha-community.org>
> Subject: Re: [Koha-devel] Hide records on Leader 05 = d in OPAC
> 
> Hi,
> 
> On Sun, Jan 10, 2016 at 8:44 PM, David Cook <dcook at prosentient.com.au>
> wrote:
> > We can’t necessarily rely on all Koha instances running this cronjob,
> > nor can we rely on the frequency. Shouldn’t we be hiding these records
> > from the OPAC as soon as they’re marked as “deleted”?
> >
> > I’ve opened a bug for this purpose:
> > http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537
> 
> I am in mild disfavor of this proposal, particularly as implemented in current
> patch. Using a cronjob to delete records where Leader/05 is set to 'd' is
> useful when the library has arranged their workflow such that they *know*
> that Leader/05 = 'd' is being used consistently to signify that a record is no
> longer wanted. However, for a library that has not hitherto cared about the
> values in that position, unconditionally suppressing the display of such
> records could come as an unwelcome surprise.
> 
> That said, it is also a reasonable choice for a library to want to use the
> Leader/05 as suppression criterion.  Consequently, I suggest adding a
> configuration option.  For that matter, making it configurable (say, by
> allowing the library to specify a set of query additions for the purpose of
> filtering records from public display) could result in a more generally useful
> mechanism.
> 
> > I admit that I have a special interest in this where I might be
> > overlaying existing records using a mostly empty skeleton  record
> > generated from an OAI-PMH identifier and a OAI-PMH deleted status
> > (OAI-PMH doesn’t send metadata for deleted records).
> >  I’d match the existing record in Koha using the identifier, and then
> > set LDR05 to “d” in accordance with the OAI-PMH deleted status. Then,
> > that record would disappear from the OPAC, so that end users don’t see
> > this skeleton record.
> 
> I do not find this a compelling use case as stated.  If the goal is to allow
> harvesting and overlay records from an OAI-PMH provider to also delete bibs
> from a Koha database... coding so that the records are
> *actually* deleted seems more direct.
> 
> Regards,
> 
> Galen
> --
> Galen Charlton
> Infrastructure and Added Services Manager Equinox Software, Inc. / The
> Open Source Experts
> email:  gmc at esilibrary.com
> direct: +1 770-709-5581
> cell:   +1 404-984-4366
> skype:  gmcharlt
> web:    http://www.esilibrary.com/
> Supporting Koha and Evergreen: http://koha-community.org &
> http://evergreen-ils.org