Record indexer integration (Zebra)
An effort is being made to update the INSTALL.* files and to aid our users in running Koha almost auto-magically (read .deb install method, copy and paste instructions for src install in debian/ubuntu, etc). A missing piece (from my point of view) is the integration of the record indexer daemon into Koha (currently the default is to reindex -z every 5', and there are links to using Koha::Contrib::Tamil tools for having a *daemon* to do it). I vote for doing an effort to have Koha::Contrib::Tamil integrated into Koha. Once it is in the source tree this will: - Avoid Koha version-specific incompatibilities (read ease of maintenance). - Provide a simple way (SysV) to run the daemon. - Make it possible for the daemon to evolve (not sure, but I read that jcamins had some patches to adapt the Koha::Contrib::Tamil to the koha-common package multi-instance schema). Even if we move to a SolR/Zebra/Whatever: if we support Zebra it will be needed. As spoken with FrédericD, this implementation of the indexer lacks some features (?) as record sanitization, or ISO record indexing instead of MARCXML, but maybe we conclude it is not needed anymore there. Or it could be reintroduced once is in the source tree. As I see it, once it is integrated in the Koha source its easier to patch and make it evolve. Regards To+
Hi all, Just for information: We (at BibLibre) have developped an indexer daemon (named IndexRecordQueue). It's available on our git repository (branch dev/solr). Direct link: http://git.biblibre.com/?p=koha;a=blob;f=misc/solr/IndexRecordQueue.pl;h=9f1... I can do some doc if someone is interested. Regards, Jonathan 2012/7/25 Tomas Cohen Arazi <tomascohen@gmail.com>
An effort is being made to update the INSTALL.* files and to aid our users in running Koha almost auto-magically (read .deb install method, copy and paste instructions for src install in debian/ubuntu, etc).
A missing piece (from my point of view) is the integration of the record indexer daemon into Koha (currently the default is to reindex -z every 5', and there are links to using Koha::Contrib::Tamil tools for having a *daemon* to do it).
I vote for doing an effort to have Koha::Contrib::Tamil integrated into Koha. Once it is in the source tree this will:
- Avoid Koha version-specific incompatibilities (read ease of maintenance). - Provide a simple way (SysV) to run the daemon. - Make it possible for the daemon to evolve (not sure, but I read that jcamins had some patches to adapt the Koha::Contrib::Tamil to the koha-common package multi-instance schema). Even if we move to a SolR/Zebra/Whatever: if we support Zebra it will be needed.
As spoken with FrédericD, this implementation of the indexer lacks some features (?) as record sanitization, or ISO record indexing instead of MARCXML, but maybe we conclude it is not needed anymore there. Or it could be reintroduced once is in the source tree.
As I see it, once it is integrated in the Koha source its easier to patch and make it evolve.
Regards To+ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi Jonathan, Il 26/07/2012 11:32, Jonathan Druart ha scritto:
Hi all,
Just for information: We (at BibLibre) have developped an indexer daemon (named IndexRecordQueue).
It's available on our git repository (branch dev/solr).
I can do some doc if someone is interested.
this script is only for solr or do you think can be used also for Zebra ? Bye Zeno -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
Dear Members, I have question regarding the content in the field 005 ( MARC tag). Is this the field in which issue and return date of a document is recorded or date on which the record was modified/updated/corrected ? There is another field 008, which stores the date on which the recorded was added to the database. pl clarify. thanks in advance, Dr.G.K.Manjunath Chief Librarian Indira Gandhi Inst of Development Research Film City Road Santosh Nagar Goregaon(East ) MUMBAI - 400 065 India Phone: 022-28416528 email: gkm@igidr.ac.in
This might explain: http://www.loc.gov/marc/bibliographic/bd005.html Doug -----Original Message----- From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of G.K.MANJUNATH Sent: Thursday, July 26, 2012 3:48 AM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] What is Trasaction Date and Time in MARC field 005 Dear Members, I have question regarding the content in the field 005 ( MARC tag). Is this the field in which issue and return date of a document is recorded or date on which the record was modified/updated/corrected ? There is another field 008, which stores the date on which the recorded was added to the database. pl clarify. thanks in advance, Dr.G.K.Manjunath Chief Librarian Indira Gandhi Inst of Development Research Film City Road Santosh Nagar Goregaon(East ) MUMBAI - 400 065 India Phone: 022-28416528 email: gkm@igidr.ac.in _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Dear Doug, Thanks for your help. In fact, I had seen the documentation available at the LOC site before sending my query. Therefore, my doubt still persists. The documentation does not clearly indicate whether the date of transaction is the date on which the record was modified/updated or this is the date on which circulation related transaction was made. I solicit clarifications from the members of the forum. thanks, G.K.Manjunath Chief Librarian IGIDR, Mumbai, India On Thu, 26 Jul 2012, Doug Dearden wrote:
This might explain: http://www.loc.gov/marc/bibliographic/bd005.html
Doug
-----Original Message----- From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of G.K.MANJUNATH Sent: Thursday, July 26, 2012 3:48 AM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] What is Trasaction Date and Time in MARC field 005
Dear Members,
I have question regarding the content in the field 005 ( MARC tag).
Is this the field in which issue and return date of a document is recorded or date on which the record was modified/updated/corrected ?
There is another field 008, which stores the date on which the recorded was added to the database.
pl clarify.
thanks in advance,
Dr.G.K.Manjunath Chief Librarian Indira Gandhi Inst of Development Research Film City Road Santosh Nagar Goregaon(East ) MUMBAI - 400 065 India Phone: 022-28416528 email: gkm@igidr.ac.in
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi, Il 27/07/2012 07:03, G.K.MANJUNATH ha scritto:
Thanks for your help. In fact, I had seen the documentation available at the LOC site before sending my query. Therefore, my doubt still persists.
The documentation does not clearly indicate whether the date of transaction is the date on which the record was modified/updated or this is the date on which circulation related transaction was made.
005 -> last date of modification of record (last time of update) 008 -> date of bibliografic item. Totaly disconnect with circolation. 005 is only about last update of cataloguing data. In fact now Koha doesn't update 005 automaticaly, you need to do by hand every time. Cheers Zeno Tajoli -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
In fact now Koha doesn't update 005 automaticaly, you need to do by hand every time. That was the case in the past. Note that ModBiblioMarc does update 005 when the record is saved. (Bug 5374).
Regards, Marcel
Zeno, To adapt for Zebra, I think you just have to modify the routine sub launch_indexation Currently, it calls C4::Search::Engine::Solr::IndexRecord( $recordtype, $recordids ); 2012/7/26 Zeno Tajoli <tajoli@cilea.it>
Hi Jonathan,
Il 26/07/2012 11:32, Jonathan Druart ha scritto:
Hi all,
Just for information: We (at BibLibre) have developped an indexer daemon (named IndexRecordQueue).
It's available on our git repository (branch dev/solr).
I can do some doc if someone is interested.
this script is only for solr or do you think can be used also for Zebra ?
Bye Zeno -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
Can we move Search::Engine::Solr into Koha:: namespace and create a Zebra version that implements the same API? (havent read the code yet, sorry). If there was a consensus on the way Koha::Contrib::Tamil does the indexing, the algorithms could be reused. Once we had this, we could introduce some more scripts/files or instructions so we have the indexing daemon ready for production, OOTB. I think this is the *only* obscure part in the Koha deployment, that we could have solved a while ago. No matter which approach/API is chosen. Regards To+ On Thu, Jul 26, 2012 at 7:24 AM, Jonathan Druart <jonathan.druart@biblibre.com> wrote:
Zeno, To adapt for Zebra, I think you just have to modify the routine sub launch_indexation
Currently, it calls C4::Search::Engine::Solr::IndexRecord( $recordtype, $recordids );
2012/7/26 Zeno Tajoli <tajoli@cilea.it>
Hi Jonathan,
Il 26/07/2012 11:32, Jonathan Druart ha scritto:
Hi all,
Just for information: We (at BibLibre) have developped an indexer daemon (named IndexRecordQueue).
It's available on our git repository (branch dev/solr).
I can do some doc if someone is interested.
this script is only for solr or do you think can be used also for Zebra ?
Bye Zeno -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Can we move Search::Engine::Solr into Koha:: namespace and create a Zebra version that implements the same API? (havent read the code yet, sorry).
Have a loot at Bug 8233 ( http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8233) and the wiki page : http://wiki.koha-community.org/wiki/SearchEngine_Layer_RFC Work in progress ;-) 2012/7/26 Tomas Cohen Arazi <tomascohen@gmail.com>
Can we move Search::Engine::Solr into Koha:: namespace and create a Zebra version that implements the same API? (havent read the code yet, sorry). If there was a consensus on the way Koha::Contrib::Tamil does the indexing, the algorithms could be reused. Once we had this, we could introduce some more scripts/files or instructions so we have the indexing daemon ready for production, OOTB.
I think this is the *only* obscure part in the Koha deployment, that we could have solved a while ago. No matter which approach/API is chosen.
Regards To+
On Thu, Jul 26, 2012 at 7:24 AM, Jonathan Druart <jonathan.druart@biblibre.com> wrote:
Zeno, To adapt for Zebra, I think you just have to modify the routine sub launch_indexation
Currently, it calls C4::Search::Engine::Solr::IndexRecord( $recordtype, $recordids );
2012/7/26 Zeno Tajoli <tajoli@cilea.it>
Hi Jonathan,
Il 26/07/2012 11:32, Jonathan Druart ha scritto:
Hi all,
Just for information: We (at BibLibre) have developped an indexer
daemon
(named IndexRecordQueue).
It's available on our git repository (branch dev/solr).
I can do some doc if someone is interested.
this script is only for solr or do you think can be used also for Zebra ?
Bye Zeno -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Is this going to be on master soon? On Thu, Jul 26, 2012 at 6:32 AM, Jonathan Druart <jonathan.druart@biblibre.com> wrote:
Hi all,
Just for information: We (at BibLibre) have developped an indexer daemon (named IndexRecordQueue). It's available on our git repository (branch dev/solr).
Direct link: http://git.biblibre.com/?p=koha;a=blob;f=misc/solr/IndexRecordQueue.pl;h=9f1...
I can do some doc if someone is interested.
Regards, Jonathan
2012/7/25 Tomas Cohen Arazi <tomascohen@gmail.com>
An effort is being made to update the INSTALL.* files and to aid our users in running Koha almost auto-magically (read .deb install method, copy and paste instructions for src install in debian/ubuntu, etc).
A missing piece (from my point of view) is the integration of the record indexer daemon into Koha (currently the default is to reindex -z every 5', and there are links to using Koha::Contrib::Tamil tools for having a *daemon* to do it).
I vote for doing an effort to have Koha::Contrib::Tamil integrated into Koha. Once it is in the source tree this will:
- Avoid Koha version-specific incompatibilities (read ease of maintenance). - Provide a simple way (SysV) to run the daemon. - Make it possible for the daemon to evolve (not sure, but I read that jcamins had some patches to adapt the Koha::Contrib::Tamil to the koha-common package multi-instance schema). Even if we move to a SolR/Zebra/Whatever: if we support Zebra it will be needed.
As spoken with FrédericD, this implementation of the indexer lacks some features (?) as record sanitization, or ISO record indexing instead of MARCXML, but maybe we conclude it is not needed anymore there. Or it could be reintroduced once is in the source tree.
As I see it, once it is integrated in the Koha source its easier to patch and make it evolve.
Regards To+ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi Tomas, Il 25/07/2012 15:38, Tomas Cohen Arazi ha scritto:
An effort is being made to update the INSTALL.* files and to aid our users in running Koha almost auto-magically (read .deb install method, copy and paste instructions for src install in debian/ubuntu, etc).
A missing piece (from my point of view) is the integration of the record indexer daemon into Koha (currently the default is to reindex -z every 5', and there are links to using Koha::Contrib::Tamil tools for having a *daemon* to do it).
I have done the doc and wiki modifications to insert the links to Koha::Contrib::Tamil tools. So for me is clarly of to insert them in main Koha git. Attention that it need many dependencies. Bye Zeno Tajoli -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
participants (6)
-
Doug Dearden -
G.K.MANJUNATH -
Jonathan Druart -
Marcel de Rooy -
Tomas Cohen Arazi -
Zeno Tajoli