$("#export #format").append("<option value='marcstd'>MARC (no 9xx)</option>");

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 --- Comment #1 from Jared Camins-Esakov <jcamins@cpbibliography.com> 2011-12-11 01:25:46 UTC --- Created attachment 6704 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6704 Bug 7345: Enable exporting records sans private fields Add an option for marcstd to the opac-export.pl and catalogue/export.pl scripts. This new format removes all 9XX, X9X, XX9 fields and subfield $9 (with the exception of 490 in flavours of MARC other than UNIMARC). The work is done in C4::Record::marc2marc. Testing plan: 1. Export a record in "MARC (Unicode/UTF-8)" format as a control 2. In the OPAC, run the following jQuery to add the marcstd option to the UI: 3. Export the same record in "MARC (no 9xx)" format 4. Compare the two, noticing that any subfield $9 or fields including 9 (other than 490 in flavours of MARC other than UNIMARC) have been removed -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Frère Sébastien Marie <semarie-koha@latrappe.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |semarie-koha@latrappe.fr --- Comment #2 from Frère Sébastien Marie <semarie-koha@latrappe.fr> 2011-12-11 10:59:51 UTC --- I think the patch could have a better design if UNIMARC was handle better: currently, if marcflavour is UNIMARC, "exporting records sans private fields" will result "exporting records with private fields" without notice to users... The design could be to have a hash with: - key $from_marcflavour - value a hash for function which are flavour dependent (MARC::Record->new_from_usmarc' / 'is_privatefield') And if not key, set $error to 'unimplemented'. This will permit easily extend for other marcflavour... -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 --- Comment #4 from Frère Sébastien Marie <semarie-koha@latrappe.fr> 2011-12-11 13:04:36 UTC --- (In reply to comment #3)

Frère Sébastien,

I don't use UNIMARC, but the patch is supposed to strip private fields from UNIMARC too. At least, I thought that fields containing a 9 were private fields in UNIMARC.

The only difference between MARC21/NORMARC and UNIMARC is supposed to be that field 490 is left in for flavors other than UNIMARC.

Can you provide more details on exactly which fields are being left in, so that I can troubleshoot it?

Sorry, I have too quickly read your code... But as you say UNIMARC is supported, I will test it. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

$("#export #format").append("<option value='marcstd'>MARC (no 9xx)</option>");

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6707|0 |1 is obsolete| | --- Comment #6 from Chris Cormack <chris@bigballofwax.co.nz> 2011-12-13 08:28:44 UTC --- Created attachment 6738 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6738 Bug 7345: Enable exporting records sans private fields Add an option for marcstd to the opac-export.pl and catalogue/export.pl scripts. This new format removes all 9XX, X9X, XX9 fields and subfield $9 (with the exception of 490 in flavours of MARC other than UNIMARC). The work is done in C4::Record::marc2marc. Testing plan: 1. Export a record in "MARC (Unicode/UTF-8)" format as a control 2. In the OPAC, run the following jQuery to add the marcstd option to the UI: 3. Export the same record in "MARC (no 9xx)" format 4. Compare the two, noticing that any subfield $9 or fields including 9 (other than 490 in flavours of MARC other than UNIMARC) have been removed Signed-off-by: Frère Sébastien Marie <semarie-koha@latrappe.fr> Test done under UNIMARC flavour. All seems to work well. Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Test done under MARC21 flavour, works fine. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.poulain@biblibre.com Patch Status|Signed Off |Failed QA --- Comment #7 from Paul Poulain <paul.poulain@biblibre.com> 2011-12-15 17:28:20 UTC --- QA comment: I must be dumb, but: the test plan include some jquery. this is OK for testing. But in real life, how will we control it we export 9 or no ? marking failed QA until I understand, switch back to signed-off once you've answered -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

Paul,

(In reply to comment #7)

...

QA comment: I must be dumb, but: the test plan include some jquery. this is OK for testing. But in real life, how will we control it we export 9 or no ?

No, you're not dumb. There is no user interface change. OK, but I still don't see what to do with this patch: if it changes nothing

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 --- Comment #9 from Paul Poulain <paul.poulain@biblibre.com> 2011-12-16 09:24:33 UTC --- (In reply to comment #8) practically, why applying it ? I was expecting a syspref for example, to decide wether to export or no 9 fields. Or some checkbox on opac-export script, to let user decide. Or something like that. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> 2011-12-16 13:06:35 UTC --- Hi Jared and Paul, perhaps this patch could go in as a first step and we could argue about how to add it to the gui later on? I would like a way to export MARC without local fields as we use some of them and an easy way to strip them out would be good. I think it could be an option for the staff interface and perhaps a system preference for the OPAC? -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch Status|Signed Off |Failed QA --- Comment #13 from Paul Poulain <paul.poulain@biblibre.com> 2011-12-30 09:45:52 UTC --- Jared, I just spoke of this entry with cait on IRC. I mark this bug "failed QA" until there is a practical & easy to use way to activate this feature. My first thoughts would be: * add a syspref to define the "local field behaviour": let the user choose to upload them or no / remove them automatically * depending on the syspref, add a checkbox on the export screen or no * depending on the syspref and/or the user choice, remove local fields before exporting or no. PS: the items are also stored in a 9xx field (995 in UNIMARC, 952 in MARC21). I strongly think we must let user export those fields (with an option not to have them ?), otherwise, they will never have items ! PS2: cait told me: <cait> perhaps I can write a follow up adding the option to the gui then a few minuts later: <cait> paul_p: I will try to work on a follow up -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6738|0 |1 is obsolete| | --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> 2012-01-01 13:27:04 UTC --- Created attachment 7001 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7001 Bug 7392: Add system preference to define export options in OPAC NOTE: Patch depends on changes made for bug 7345 for the "marcstd" option. It adds a new system preference 'OpacExportOptions': The new system preference allows to give a list of | separated values to define, which download options are offered in OPAC. Possible options are listed in the system preference editor and include the new export option 'marcstd' from bug 7345. The sequence of the download options in the system preference will be the sequence in the OPAC pull down. If left empty, the export option will be hidden from OPAC. This patch creates a new include file for the OPAC with all the code that is common to the three detail views (regular, MARC, and ISBD). As a side effect of creating this include file, some inconsistencies in OpacSearchForTitleIn were fixed. {BIBLIONUMBER} and {CONTROLNUMBER} were broken for ISBD and MARC view before and should now be replaced with the correct values in all tabs. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6738|1 |0 is obsolete| | -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch Status|Failed QA |Needs Signoff -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch Status|Needs Signoff |Signed Off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl Patch Status|Signed Off |Failed QA --- Comment #18 from M. de Rooy <m.de.rooy@rijksmuseum.nl> 2012-01-13 15:52:32 UTC --- QA Comment: This looks VERY promising! Some points to note: CONFLICT (content): Merge conflict in installer/data/mysql/sysprefs.sql Auto-merged installer/data/mysql/updatedatabase.pl CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl Resolved them but unfortunately had more remarks ;) In the pref file I find a template variable: Use the CSS stylesheet [% opacthemelang %]/css/ This does not work here. OpacExportOptions: It does not really read comfortable in the Preferences screen. All options exceed the length of the field. Could you extend the field, make it better readable (new line)? [Is no blocker but a design thing..] MARC standard is promised but not delivered yet (while pending on other report). Does not block this, but you could move it to the report it belongs.. Just a thought: If you add an unexisting format, it now comes in the list and you will have an error message on screen. Would it be interesting to exclude these formats (question with some doubts on my part too).. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 --- Comment #20 from Katrin Fischer <katrin.fischer@bsz-bw.de> 2012-01-13 16:07:44 UTC --- Hi Marcel, thx for testing this :)

In the pref file I find a template variable: Use the CSS stylesheet [% opacthemelang %]/css/ This does not work here.

I changed that from the old template syntax to TT - so it probably never worked. Not sure how to make it work? I can take the line change out - but it will still be broken.

OpacExportOptions: It does not really read comfortable in the Preferences screen. All options exceed the length of the field. Could you extend the field, make it better readable (new line)? [Is no blocker but a design thing..]

It seems the appearance of the field is determined by the type you give in the pref file. I am not sure there is a better type we can use here. The only other option seems to be a box like we use for OpacUserCSS and others?

MARC standard is promised but not delivered yet (while pending on other report). Does not block this, but you could move it to the report it belongs.

I have marked both bugs as dependent on each other. The only thing keeping 7392 from being pushed is the missing presence in the GUI. So I thought I would look into it and it turned into a new feature, going beyond a simple follow-up.

Just a thought: If you add an unexisting format, it now comes in the list and you will have an error message on screen. Would it be interesting to exclude these formats (question with some doubts on my part too)..

I can make them not show up - that's probably the easiest way to solve this. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch Status|Failed QA |Signed Off --- Comment #22 from M. de Rooy <m.de.rooy@rijksmuseum.nl> 2012-01-13 18:07:21 UTC --- Restoring patch status.. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #23 from M. de Rooy <m.de.rooy@rijksmuseum.nl> 2012-02-03 09:27:33 UTC --- QA Comment: First, looks very good, but still have a few questions: 1) Will a user know that marcstd means marc without private fields (9XX, X9X and XX9)? I did not realize it rightaway too ;) BTW Does not block this patch! 2) While testing it, I saw that the fields 952 and 999 are still included. Note that these are certainly no marc standard fields, so if any should be excluded, they should. Since your regex tests /9/, I do however not understand why they are still there. Do you? Note that a local field 942 and subfields 9 were correctly removed in my testing. Could you test this too? 3) I doubt if the encoding is 100% correct. When I opened the exported file in notepad, some characters with umlaut did not come up completely correct. But when copying them somewhere else, they were correct after all. When I commented the call to SetUTFFlag, the results in notepad were correct. Tested with title: Vorläufer, Schüler, Zeitgenossen. Please note that GetMarcBiblio is already calling _new_from_xml with a utf8 parameter. Also note that this remark also pertains to existing format utf8 in export scripts. (So formally, it could be a report on itself.) Maybe Katrin can test it (probably she has the most experience with umlauts ;) 4) Finally, elaborating on point 3: Not blocking this patch, but why would the marc2marc function not just demand a marc object (instead of optionally creating it). Note that the only call now already receives an object, created by GetMarcBiblio. And why not leave the call to as_usmarc to the export script, just as done in the case of format marc8, utf8 ? Changing status just to reflect need for some testing and answers.. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 --- Comment #25 from M. de Rooy <m.de.rooy@rijksmuseum.nl> 2012-02-03 09:48:42 UTC --- (In reply to comment #24)

I don't think the user is seeing this somewhere? I used the codes for my syspref (bug 7392) but tried to explain in the text of the pref. So perhaps we should change the explanation / name shown in OPAC if it's not clear?

Yes, maybe Jared could come up with a "better" description? -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 --- Comment #27 from Katrin Fischer <katrin.fischer@bsz-bw.de> 2012-02-03 09:54:47 UTC --- I think as the item format is Koha specific it will not be useful for patrons, so perhaps the OPAC should not offer this information. Also - will the download include fields like internal notes? Because if it does... this is another problem. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 --- Comment #29 from Jared Camins-Esakov <jcamins@cpbibliography.com> 2012-02-05 20:01:13 UTC --- (In reply to comment #23)

QA Comment: First, looks very good, but still have a few questions:

1) Will a user know that marcstd means marc without private fields (9XX, X9X and XX9)? I did not realize it rightaway too ;) BTW Does not block this patch!

I still argue that this is primarily a developer-centric patch. It's needed so that we can interoperate with the rest of the library world who do not speak Koha. If you have a better idea for the label, please submit a follow-up patch by all means. I do not have any better ideas. I would probably use jQuery to call it "Raw MARC" because I don't see any use for the existing UTF-8 and MARC-8 exports.

2) While testing it, I saw that the fields 952 and 999 are still included. Note that these are certainly no marc standard fields, so if any should be excluded, they should. Since your regex tests /9/, I do however not understand why they are still there. Do you? Note that a local field 942 and subfields 9 were correctly removed in my testing. Could you test this too?

I did not have this problem, but your proposed fix worked, so I will attach the revised patch momentarily.

3) I doubt if the encoding is 100% correct. When I opened the exported file in notepad, some characters with umlaut did not come up completely correct. But when copying them somewhere else, they were correct after all. When I commented the call to SetUTFFlag, the results in notepad were correct. Tested with title: Vorläufer, Schüler, Zeitgenossen. Please note that GetMarcBiblio is already calling _new_from_xml with a utf8 parameter. Also note that this remark also pertains to existing format utf8 in export scripts. (So formally, it could be a report on itself.) Maybe Katrin can test it (probably she has the most experience with umlauts ;)

I think it is better not to fix this here, because I am not an expert in this, and it would be better to have someone who understands the problem fix it in both places than have me try to hack in a fix here.

4) Finally, elaborating on point 3: Not blocking this patch, but why would the marc2marc function not just demand a marc object (instead of optionally creating it). Note that the only call now already receives an object, created by GetMarcBiblio. And why not leave the call to as_usmarc to the export script, just as done in the case of format marc8, utf8 ?

I was using the existing marc2marc API, which specified that it should return the results of as_usmarc.

Changing status just to reflect need for some testing and answers..

-- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #32 from M. de Rooy <m.de.rooy@rijksmuseum.nl> 2012-02-06 07:46:14 UTC --- QA Comment: With reference to comment 23. Point 1. Name may need a followup. Note BTW that this patch needs 7392 to make the option visible. For testing purposes I manually added a line to opac-results.tt to add this option. Point 2. Most important one: Solved. Note that this may depend too on the version of MARC::Record. The problem of delete_fields may have been resolved in a newer version. Point 3. Still needs attention, but does not block this patch. This already occurs in current code (option utf8). Point 4. May not have been fully understood. Does not matter to me. Only theory. Marking it as Passed QA. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Failed QA --- Comment #34 from Paul Poulain <paul.poulain@biblibre.com> 2012-02-10 15:16:25 UTC --- oups, wrong status sorry -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7566|0 |1 is obsolete| | --- Comment #36 from M. de Rooy <m.de.rooy@rijksmuseum.nl> 2012-02-13 08:06:02 UTC --- Created attachment 7612 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7612 Patch Amended Jared's patch to resolve double defition of $error in export.pl -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7345 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master Version|master |rel_3_8 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.