- Koha-bugs - lists.koha-community.org

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 David Cook <dcook(a)prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #81783|0 |1 is obsolete| | --- Comment #237 from David Cook <dcook(a)prosentient.com.au> --- Created attachment 81861 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81861&action=ed… Bug 10662: Build OAI-PMH Harvesting Client This patch adds an OAI-PMH harvesting client to Koha. The client runs as a daemon in the background. Users interact with the client via the Koha web user interface, which communicates with the daemon via a unix socket using a simple JSON-based protocol. The harvester ingests MARCXML. You can harvest other metadata formats, but you must use a XSLT to transform them into MARCXML, if you want them to be imported into Koha. You can supply your own download and import modules via the oai-pmh-harvester.yaml configuration file, but the default modules supplied in this patch should be good enough for your purposes. If they're not, raise a Bugzilla issue. There is a cleanup_database.pl addition, because high volume harvesting will cause the oai_harvester_import_queue table to fill quickly. This table is not required for adding/updating records. It's mostly just for general monitoring and audit purposes. Signed-off-by: Andreas Hedström Mace <andreas.hedstrom.mace(a)sub.su.se> -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #236 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #224) > Comment on attachment 81789 [details] [review] > Bug 10662: (QA follow-up) Fix plural in pod and use statements > > Review of attachment 81789 [details] [review]: > ----------------------------------------------------------------- > > ::: Koha/OAI/Harvester/Import/Record.pm > @@ +29,4 @@ > > > > use Koha::OAI::Harvester::Import::MARCXML; > > use Koha::OAI::Harvester::Biblios; > > +use Koha::OAI::Harvester::Histories; > > This actually should be Koha::OAI::Harvester::History and not > Koha::OAI::Harvester::Histories. This change breaks the harvester. Actually, since Histories uses History, it's probably fine... but I don't actually use Histories in Record.pm. -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #235 from David Cook <dcook(a)prosentient.com.au> --- (In reply to Josef Moravec from comment #209) > Created attachment 81786 [details] > Encoding problem + datatable > > In submitted requests table I encountered a encoding problem: > > "Knihovna ÃstÃ" should be "Knihovna Ústí". > > Also, the heading of datatable is usually formatted in one line, see patron > circulation history for example. Ok I've fixed that. I was reversing the Encode module encode/decode functions and not reading the output of Devel::Peek correctly. Now I can see both Knihovna Ústí and 我不好 coming back from the harvester as well. -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #234 from David Cook <dcook(a)prosentient.com.au> --- Yep... put in some Chinese characters and now that decode isn't working either. Let's try this again... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #233 from David Cook <dcook(a)prosentient.com.au> --- (In reply to Josef Moravec from comment #209) > Created attachment 81786 [details] > Encoding problem + datatable > > In submitted requests table I encountered a encoding problem: > > "Knihovna ÃstÃ" should be "Knihovna Ústí". > > Also, the heading of datatable is usually formatted in one line, see patron > circulation history for example. I am stumped by this one. I've added a Encode::Decode("UTF-8",$json_message") to the client used by the web page, and that gets the characters to render as Knihovna Ústí on the web page... but when I look at the actual hex code in the variable... it's not valid UTF8. It's Windows 1252/Latin-1. The hex is 4b6e69686f766e6120da7374ed, and the internal representation in Perl is PV = 0xa6bce80 "Knihovna \303\232st\303\255"\0 [UTF8 "Knihovna \x{da}st\x{ed}"]. So DA and ED are Unicode code points that match Ú and í (https://www.utf8-chartable.de/unicode-utf8-table.pl) However, DA and ED are also the Latin-1 hex for Ú and í (https://en.wikipedia.org/wiki/Windows-1252)(https://en.wikipedia.org/wiki/I…) I think maybe I need to try with some Chinese characters that don't exist in Latin-1... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #232 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #226) > Interesting... > > I can't even submit a task with a name of "Knihovna Ústí". The harvester is > failing to add it. I'll look into it. Ah, I see the problem I was having. Just me being silly. Now I see the problem. I have some ideas with this one... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #231 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #228) > (In reply to David Cook from comment #227) > > I think KohaTable is breaking some functionality from DataTables, because I > > should be able to do this on a KohaTable: > > https://datatables.net/reference/api/ajax.reload() > > I think KohaTable is using the legacy API and DataTables allowed me to use > the current API. > > Uploading screenshots to show the different objects returned by DataTable > and KohaTable... OK I figured it out. When using "$( selector ).DataTable();", the return value is the API object. When using KohaTable, the return value is the jQuery object. So I just needed to add .dataTable().api() to the back of the jQuery object to access the DataTable API. -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #230 from David Cook <dcook(a)prosentient.com.au> --- Created attachment 81856 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81856&action=ed… Object returned by DataTable -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #229 from David Cook <dcook(a)prosentient.com.au> --- Created attachment 81855 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=81855&action=ed… Object returned by KohaTable -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 10662] Build OAI-PMH Harvesting Client
by bugzilla-daemon＠bugs.koha-community.org 02 Nov '18

02 Nov '18

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 --- Comment #228 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #227) > I think KohaTable is breaking some functionality from DataTables, because I > should be able to do this on a KohaTable: > https://datatables.net/reference/api/ajax.reload() I think KohaTable is using the legacy API and DataTables allowed me to use the current API. Uploading screenshots to show the different objects returned by DataTable and KohaTable... -- You are receiving this mail because: You are watching all bug changes.

1 0