Bulk Marc Import via the Web
Hi all I am discussing the possibility of wrapping the bulkmarcimport.pl script in a web page with Bob Birchall. The page would offer a form that contains the command line options as checkboxes or textboxes as appopriate along with a file input control to upload a .MRC file. Upon completion of the form, the web page validates the .MRC as far as that is possible (hints anyone?) and then passes it on to bulkmarcimport.pl. The output from the script is captured and sent back to the the web user. Seems simple enough. The only problem I imagine encountering is the web browser timing out for an import that takes a long time. Any advice and observations from the developers is most welcome. cheers rickw -- _________________________________ Rick Welykochy || Praxis Services Politics is the business of getting power and privilege without possessing merit. -- PJ O'Rourke
On 26 April 2010 23:29, Rick Welykochy <rick@praxis.com.au> wrote:
Hi all
I am discussing the possibility of wrapping the bulkmarcimport.pl script in a web page with Bob Birchall.
The page would offer a form that contains the command line options as checkboxes or textboxes as appopriate along with a file input control to upload a .MRC file.
Upon completion of the form, the web page validates the .MRC as far as that is possible (hints anyone?) and then passes it on to bulkmarcimport.pl. The output from the script is captured and sent back to the the web user.
Seems simple enough. The only problem I imagine encountering is the web browser timing out for an import that takes a long time.
Any advice and observations from the developers is most welcome.
Hi Rick There is already a bulk marc importer in the web interface. A 2 step process, 1st step stages it, the second imports it. Can't you just use that? Chris
Chris Cormack wrote:
There is already a bulk marc importer in the web interface. A 2 step process, 1st step stages it, the second imports it. Can't you just use that?
Goodness gracious me. Yup, found it in Tools. I will pass this on to Bob and he can give it a try with some data. Thanks for the heads up :) Bob: see Tools > Stage MARC Records For Import cheers rickw -- _________________________________ Rick Welykochy || Praxis Services Politics is the business of getting power and privilege without possessing merit. -- PJ O'Rourke
What we need a web interface for is the bulkauthimport ... if you still want to work on something :) Nicole On Mon, Apr 26, 2010 at 7:46 AM, Rick Welykochy <rick@praxis.com.au> wrote:
Chris Cormack wrote:
There is already a bulk marc importer in the web interface. A 2 step process, 1st step stages it, the second imports it. Can't you just use that?
Goodness gracious me. Yup, found it in Tools. I will pass this on to Bob and he can give it a try with some data. Thanks for the heads up :)
Bob: see Tools > Stage MARC Records For Import
cheers rickw
-- _________________________________ Rick Welykochy || Praxis Services
Politics is the business of getting power and privilege without possessing merit. -- PJ O'Rourke _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Chris, My understanding of that tool is that it does not import holdings data. Only bulkmarcimport.pl does that. Am I right? Bob
-----Original Message----- From: Rick Welykochy [mailto:rick@praxis.com.au] Sent: Monday, 26 April 2010 9:47 PM To: Chris Cormack Cc: koha-devel@lists.koha.org; Bob Birchall Subject: Re: [Koha-devel] Bulk Marc Import via the Web
Chris Cormack wrote:
There is already a bulk marc importer in the web interface. A 2 step process, 1st step stages it, the second imports it. Can't you just use that?
Goodness gracious me. Yup, found it in Tools. I will pass this on to Bob and he can give it a try with some data. Thanks for the heads up :)
Bob: see Tools > Stage MARC Records For Import
cheers rickw
-- _________________________________ Rick Welykochy || Praxis Services
Politics is the business of getting power and privilege without possessing merit. -- PJ O'Rourke
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.437 / Virus Database: 271.1.1/2835 - Release Date: 04/25/10 18:31:00
On 27 April 2010 00:01, Bob Birchall @ Calyx <bob@calyx.net.au> wrote:
Chris, My understanding of that tool is that it does not import holdings data. Only bulkmarcimport.pl does that. Am I right?
No, it can import item information also. Its one of the options (create items or replace existing iirc) Chris
Hello, You also might be interested to know that Kyle Hall is developing a templating system for MARC imports for CCFLS, that can be used in conjunction with the existing staged marc record management tool to map the fields in a marc file to where you want them to be in Koha. He's (I would guess) about halfway done writing it, and will be contributing it soon. I figure we'd better let the community know about it, because I'm sure others will find it useful! Cheers! Cindy ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Cindy Murdock Ames IT Services Director Meadville Public Library | CCFLS http://meadvillelibrary.org | http://ccfls.org Rick Welykochy wrote:
Hi all
I am discussing the possibility of wrapping the bulkmarcimport.pl script in a web page with Bob Birchall.
The page would offer a form that contains the command line options as checkboxes or textboxes as appopriate along with a file input control to upload a .MRC file.
Upon completion of the form, the web page validates the .MRC as far as that is possible (hints anyone?) and then passes it on to bulkmarcimport.pl. The output from the script is captured and sent back to the the web user.
Seems simple enough. The only problem I imagine encountering is the web browser timing out for an import that takes a long time.
Any advice and observations from the developers is most welcome.
cheers rickw
It's about 90% done now, but I have to wait for my mind to return before I can finish it ; ) Screenshot: http://imgur.com/9VZKK.jpg Kyle http://www.kylehall.info Information Technology Crawford County Federated Library System ( http://www.ccfls.org ) On Mon, Apr 26, 2010 at 10:21 AM, Cindy Murdock Ames <cmurdock@ccfls.org> wrote:
Hello,
You also might be interested to know that Kyle Hall is developing a templating system for MARC imports for CCFLS, that can be used in conjunction with the existing staged marc record management tool to map the fields in a marc file to where you want them to be in Koha. He's (I would guess) about halfway done writing it, and will be contributing it soon. I figure we'd better let the community know about it, because I'm sure others will find it useful!
Cheers!
Cindy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Cindy Murdock Ames IT Services Director Meadville Public Library | CCFLS http://meadvillelibrary.org | http://ccfls.org
Rick Welykochy wrote:
Hi all
I am discussing the possibility of wrapping the bulkmarcimport.pl script in a web page with Bob Birchall.
The page would offer a form that contains the command line options as checkboxes or textboxes as appopriate along with a file input control to upload a .MRC file.
Upon completion of the form, the web page validates the .MRC as far as that is possible (hints anyone?) and then passes it on to bulkmarcimport.pl. The output from the script is captured and sent back to the the web user.
Seems simple enough. The only problem I imagine encountering is the web browser timing out for an import that takes a long time.
Any advice and observations from the developers is most welcome.
cheers rickw
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
I've also got a working patch which targets MARC21. I've used it for migration and importing from vendor supplied marc files. It's not as nice or flexible as what Kyle is working on though, so I won't recommend it for inclusion in the general code tree at this point. It would work in a pinch though if you need something right away. On Mon, 2010-04-26 at 10:21 -0400, Cindy Murdock Ames wrote:
Hello,
You also might be interested to know that Kyle Hall is developing a templating system for MARC imports for CCFLS, that can be used in conjunction with the existing staged marc record management tool to map the fields in a marc file to where you want them to be in Koha. He's (I would guess) about halfway done writing it, and will be contributing it soon. I figure we'd better let the community know about it, because I'm sure others will find it useful!
Cheers!
Cindy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Cindy Murdock Ames IT Services Director Meadville Public Library | CCFLS http://meadvillelibrary.org | http://ccfls.org
Rick Welykochy wrote:
Hi all
I am discussing the possibility of wrapping the bulkmarcimport.pl script in a web page with Bob Birchall.
The page would offer a form that contains the command line options as checkboxes or textboxes as appopriate along with a file input control to upload a .MRC file.
Upon completion of the form, the web page validates the .MRC as far as that is possible (hints anyone?) and then passes it on to bulkmarcimport.pl. The output from the script is captured and sent back to the the web user.
Seems simple enough. The only problem I imagine encountering is the web browser timing out for an import that takes a long time.
Any advice and observations from the developers is most welcome.
cheers rickw
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
Hi all, Our librarian is having trouble with her MARC record imports. On some of the imports, the record imports, but doesn't import the item. So in an OPAC search the record appears, but shows "Availability: No items available" The librarian has worked with the vendors to modify MARC records, but it's a hit-or-miss situation where some records will update with items and others won't. I just performed a zebra reindex last week for another issue and that didn't help with this issue. I assume the MARC record needs specific information in a specific location to properly map to the Koha database during the import process. Is that an accurate assumption? Can anyone tell me where in a MARC record the item information needs to be stored so I can assist my librarian? She is currently having to manually add items in Koha to books that are on the shelf when students are checking out the book. Any and all help is very much appreciated :--D Thanks, Rick ------------------------------------------- Rick Forristall Programmer Analyst Goodwill of Central AZ 602.535.4078 ------------------------------------------- Through your generous material and cash donations - we provide free human services programs that create hope, jobs and sustainability for central Arizonans. In 2009 your donations transformed 25,000 lives. Good Stuff. Good Work. Goodwill. We Put People to Work! A 2009 Best Places to Work Winner www.goodwillaz.org
Richard, This is not a development question and might have been better on the general list. Koha stores item information in the 952 tag of the marc record. So the first question may be whether all the records have a 952 tag. The fields within that tag can be readily understood by viewing 'Koha to Marc Mapping' on the Administration menu and selecting 'items' from the drop down. (Apologies if this is more basic information than you need.) Do you have root access? If so, simply count the biblio, biblioitems and the items tables and see if you have a reasonable correspondence. (You may have more items than biblios if there are multiple holdings of particular titles.) To go further, you could export (part of) your catalogue using the Export Bibliographic and Holdings utility on the Tools menu, then use MarcEdit (a Windows application, sadly) to break your marc file into human readable form. Do all the items have a barcode (952$p ; items.barcode), a home branch (952$a ; items.homebranch) and a holding branch (952$b ; items.holdingbranch)? I assume your problem manifests in the staff client, not just the OPAC? Hope that helps, Bob Birchall CALYX
-----Original Message----- From: koha-devel-bounces@lists.koha.org [mailto:koha-devel-bounces@lists.koha.org] On Behalf Of Richard Forristall Sent: Wednesday, 28 April 2010 8:04 AM To: koha-devel@lists.koha.org Subject: [Koha-devel] MARC Record Import - Not Getting Items
Hi all,
Our librarian is having trouble with her MARC record imports. On some of the imports, the record imports, but doesn't import the item. So in an OPAC search the record appears, but shows "Availability: No items available"
The librarian has worked with the vendors to modify MARC records, but it's a hit-or-miss situation where some records will update with items and others won't.
I just performed a zebra reindex last week for another issue and that didn't help with this issue.
I assume the MARC record needs specific information in a specific location to properly map to the Koha database during the import process. Is that an accurate assumption?
Can anyone tell me where in a MARC record the item information needs to be stored so I can assist my librarian?
She is currently having to manually add items in Koha to books that are on the shelf when students are checking out the book.
Any and all help is very much appreciated :--D
Thanks,
Rick ------------------------------------------- Rick Forristall Programmer Analyst Goodwill of Central AZ 602.535.4078 -------------------------------------------
Through your generous material and cash donations - we provide free human services programs that create hope, jobs and sustainability for central Arizonans. In 2009 your donations transformed 25,000 lives. Good Stuff. Good Work. Goodwill.
We Put People to Work! A 2009 Best Places to Work Winner www.goodwillaz.org _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.437 / Virus Database: 271.1.1/2835 - Release Date: 04/27/10 06:27:00
Bob, Thanks so much for the info. As the developer solely responsible for supporting our librarian, for me it has become both a development issue and end-user issue. Not being sure if it was code modification required or just ensuring the MARC record had the requisite data, I opted to ask the development experts on this list for the advice -- thanks for your patience :--) Armed with my new knowledge, I'll dig back through some of the MARC records our librarian has downloaded and view some that are currently in the database using the export you refer to -- I grabbed a copy of Marc Edit about a month ago -- it's nice to be able to see the clear text of a Marc record :--) Thanks again for your assistance! Rick Forristall Programmer Analyst Goodwill of Central AZ www.GoodwillAZ.org -----Original Message----- From: Bob Birchall @ Calyx [mailto:bob@calyx.net.au] Sent: Tue 4/27/2010 4:38 PM To: Richard Forristall; koha-devel@lists.koha.org Subject: RE: [Koha-devel] MARC Record Import - Not Getting Items Richard, This is not a development question and might have been better on the general list. Koha stores item information in the 952 tag of the marc record. So the first question may be whether all the records have a 952 tag. The fields within that tag can be readily understood by viewing 'Koha to Marc Mapping' on the Administration menu and selecting 'items' from the drop down. (Apologies if this is more basic information than you need.) Do you have root access? If so, simply count the biblio, biblioitems and the items tables and see if you have a reasonable correspondence. (You may have more items than biblios if there are multiple holdings of particular titles.) To go further, you could export (part of) your catalogue using the Export Bibliographic and Holdings utility on the Tools menu, then use MarcEdit (a Windows application, sadly) to break your marc file into human readable form. Do all the items have a barcode (952$p ; items.barcode), a home branch (952$a ; items.homebranch) and a holding branch (952$b ; items.holdingbranch)? I assume your problem manifests in the staff client, not just the OPAC? Hope that helps, Bob Birchall CALYX
-----Original Message----- From: koha-devel-bounces@lists.koha.org [mailto:koha-devel-bounces@lists.koha.org] On Behalf Of Richard Forristall Sent: Wednesday, 28 April 2010 8:04 AM To: koha-devel@lists.koha.org Subject: [Koha-devel] MARC Record Import - Not Getting Items
Hi all,
Our librarian is having trouble with her MARC record imports. On some of the imports, the record imports, but doesn't import the item. So in an OPAC search the record appears, but shows "Availability: No items available"
The librarian has worked with the vendors to modify MARC records, but it's a hit-or-miss situation where some records will update with items and others won't.
I just performed a zebra reindex last week for another issue and that didn't help with this issue.
I assume the MARC record needs specific information in a specific location to properly map to the Koha database during the import process. Is that an accurate assumption?
Can anyone tell me where in a MARC record the item information needs to be stored so I can assist my librarian?
She is currently having to manually add items in Koha to books that are on the shelf when students are checking out the book.
Any and all help is very much appreciated :--D
Thanks,
Rick ------------------------------------------- Rick Forristall Programmer Analyst Goodwill of Central AZ 602.535.4078 -------------------------------------------
Through your generous material and cash donations - we provide free human services programs that create hope, jobs and sustainability for central Arizonans. In 2009 your donations transformed 25,000 lives. Good Stuff. Good Work. Goodwill.
We Put People to Work! A 2009 Best Places to Work Winner www.goodwillaz.org _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.437 / Virus Database: 271.1.1/2835 - Release Date: 04/27/10 06:27:00
2010/4/28 Richard Forristall <Richard.Forristall@goodwillaz.org>:
Bob,
Thanks so much for the info. As the developer solely responsible for supporting our librarian, for me it has become both a development issue and end-user issue. Not being sure if it was code modification required or just ensuring the MARC record had the requisite data, I opted to ask the development experts on this list for the advice -- thanks for your patience :--)
Armed with my new knowledge, I'll dig back through some of the MARC records our librarian has downloaded and view some that are currently in the database using the export you refer to -- I grabbed a copy of Marc Edit about a month ago -- it's nice to be able to see the clear text of a Marc record :--)
Hi Rick Another trick is the marcxml column in the biblioitems table. It is quite easily human readable. SELECT marcxml FROM biblioitems WHERE biblionumber=some number Chris
Thanks Chris, I feel like I just peeked behind the Wizard's curtain in OZ :--) (http://www.imdb.com/title/tt0032138/) With this information and Bob's help I feel like I can better assist our librarian with her needs. Thanks, Rick ------------------------------------------- Rick Forristall Programmer Analyst Goodwill of Central AZ 602.535.4078 ------------------------------------------- Through your generous material and cash donations - we provide free human services programs that create hope, jobs and sustainability for central Arizonans. In 2009 your donations transformed 25,000 lives. Good Stuff. Good Work. Goodwill. We Put People to Work! A 2009 Best Places to Work Winner www.goodwillaz.org -----Original Message----- From: Chris Cormack [mailto:chris@bigballofwax.co.nz] Sent: Tuesday, April 27, 2010 10:01 PM To: Richard Forristall Cc: Bob Birchall @ Calyx; koha-devel@lists.koha.org Subject: Re: [Koha-devel] MARC Record Import - Not Getting Items 2010/4/28 Richard Forristall <Richard.Forristall@goodwillaz.org>:
Bob,
Thanks so much for the info. As the developer solely responsible for supporting our librarian, for me it has become both a development issue and end-user issue. Not being sure if it was code modification required or just ensuring the MARC record had the requisite data, I opted to ask the development experts on this list for the advice -- thanks for your patience :--)
Armed with my new knowledge, I'll dig back through some of the MARC records our librarian has downloaded and view some that are currently in the database using the export you refer to -- I grabbed a copy of Marc Edit about a month ago -- it's nice to be able to see the clear text of a Marc record :--)
Hi Rick Another trick is the marcxml column in the biblioitems table. It is quite easily human readable. SELECT marcxml FROM biblioitems WHERE biblionumber=some number Chris
participants (8)
-
Bob Birchall ＠ Calyx -
Chris Cormack -
Cindy Murdock Ames -
Kyle Hall -
Michael Hafen -
Nicole Engard -
Richard Forristall -
Rick Welykochy