[Koha-cvs] koha/misc migration_tools/rebuild_zebra.pl zebr...

[paul poulain]

CVSROOT:	/cvsroot/koha
Module name:	koha
Branch: 	
Changes by:	paul poulain <tipaul at savannah.gnu.org>	06/02/09 10:59:34

Modified files:
	misc/migration_tools: rebuild_zebra.pl 
	misc/zebra/unimarc: zebra.cfg 
Added files:
	misc/migration_tools: rebuild_zebra_idx.pl test_cql2rpn.pl 
	misc/zebra/unimarc: collection.abs pqf.properties 

Log message:
	OK, this time it seems to work. The last blocking problem was... a space in
	recordId: (bib1,Identifier-standard) just after the comma. Adam agreed it was a bug, and it should be solved soon. But now we are aware, we can avoid putting the space !
	
	In this commit you have all what is needed to setup a working zebra DB in Unimarc :
	* collection.abs is UNIMARC specific and must be rewritten for MARC21, in marc21 directory
	* pdf.properties is to be copied unmodified in the marc21 directory (can also be put somewhere else)
	* rebuild_zebra.pl is SLOW, but 1 step reindexing tool, using ZOOM
	* rebuild_zebra_idx is FAST, but 2 step reindexing tool, and does not use zebra. run it, it will create all biblios XML files in /zebra/biblios directory, then zebraidx update biblios in your zebra directory
	* zebra.cfg is the zebra config file ;-)
	* test_cql2rpn.pl is a script that will query the database and show the results. Works for me, just change the query at the beginning to get answers you expect.
	
	What has to be done :
	* benchmarking : it seems the zebraidx update is faster than lightning (400biblios/sec : 10 000biblios in 25seconds), while ZOOM indexing is slow (something like 25biblios/second) More benchmarking could be done.
	* completing collection.abs for UNIMARC. I'll take care of it.
	* modifying Biblio.pm to use ZOOM instead of the "zebraidx through exec" running actually. I'll take care of it also.
	* modify the search API & tools & screens. I'll let the ball to someone else (chris ?) for this. I agree SearchMarc.pm can be dropped and replaced by something else (maybe a new-and-clean Search.pm package)

CVSWeb URLs:
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/migration_tools/rebuild_zebra_idx.pl?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/migration_tools/rebuild_zebra.pl.diff?tr1=1.3&tr2=1.4&r1=text&r2=text
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/migration_tools/test_cql2rpn.pl?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/zebra/unimarc/collection.abs?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/zebra/unimarc/pqf.properties?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/zebra/unimarc/zebra.cfg.diff?tr1=1.5&tr2=1.6&r1=text&r2=text

Patches:
Index: koha/misc/migration_tools/rebuild_zebra.pl
diff -u koha/misc/migration_tools/rebuild_zebra.pl:1.3 koha/misc/migration_tools/rebuild_zebra.pl:1.4
--- koha/misc/migration_tools/rebuild_zebra.pl:1.3	Tue Jan 10 17:03:32 2006
+++ koha/misc/migration_tools/rebuild_zebra.pl	Thu Feb  9 10:59:34 2006
@@ -41,11 +41,11 @@
 }
 
 # first, drop Zebra DB
-eval {
-	my $Zpackage = $Zconn->package();
-	$Zpackage->option(databaseName => 'Koha');
-# 	$Zpackage->send("drop");
-};
+# eval {
+# 	my $Zpackage = $Zconn->package();
+# 	$Zpackage->option(databaseName => 'Koha');
+# # 	$Zpackage->send("drop");
+# };
 
 eval {
 	my $Zpackage = $Zconn->package();
@@ -56,19 +56,20 @@
 unless (opendir(DIR, "$cgidir")) {
 		$cgidir = C4::Context->intranetdir."/";
 } 
-
 my $starttime = gettimeofday;
 my $sth = $dbh->prepare("select biblionumber from biblio");
 $sth->execute;
 my $i=0;
 while ((my $biblionumber) = $sth->fetchrow) {
 	my $record = XMLgetbiblio($dbh,$biblionumber);
+# 	warn "\n==============\n$record\n==================\n";
 	my $Zpackage = $Zconn->package();
 	$Zpackage->option(databaseName => 'Koha');
-	$Zpackage->option(action => "recordInsert");
+	$Zpackage->option(action => "specialUpdate");
+# 	$Zpackage->option(recordIdNumber => $biblionumber);
 	$Zpackage->option(record => $record);
 	$Zpackage->send("update");
-	$Zpackage->destroy;
+# 	$Zpackage->destroy;
 	$i++;
 	print '.';
 	print "$i\r" unless ($i % 100);
Index: koha/misc/zebra/unimarc/zebra.cfg
diff -u koha/misc/zebra/unimarc/zebra.cfg:1.5 koha/misc/zebra/unimarc/zebra.cfg:1.6
--- koha/misc/zebra/unimarc/zebra.cfg:1.5	Tue Jan 10 17:01:55 2006
+++ koha/misc/zebra/unimarc/zebra.cfg	Thu Feb  9 10:59:34 2006
@@ -1,18 +1,15 @@
-# Simple Zebra configuration file that defines
-# a database with USMARC records.
-# $Id: zebra.cfg,v 1.5 2006/01/10 17:01:55 tipaul Exp $
-#
 # Where are the config files located?
 profilePath: ${srcdir:-.}:/usr/local/share/idzebra/tab/
 
+
 # Files that describe the attribute sets supported.
 attset: bib1.att
 attset: explain.att
 
-recordId: (bib1,Local-number) 
+recordId: (bib1,Identifier-standard)
 
 # Specify record type
-recordType: grs.marc.unimarc
+recordType: grs.xml
 
 # Lock File Area
 lockDir: lock
@@ -27,7 +24,7 @@
 memMax: 100
 
 perm.anonymous: rw 
-encoding utf8
+encoding utf-8
 
 storeKeys:1
 storeData:1