http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10488 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=11586 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=11030 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=11330 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- After reading bug 6007, I think perhaps it would be a good idea to have 'Display only used Tags/Subfields' on by default (provided that it's fixed/accurate). That could solve the potential problem of unused fields/subfields showing up when library staff are editing their existing frameworks (where they might have deleted a lot of standard fields/subfields). -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10881 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=9678 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=1289 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Sorry for my delayed response! Competing priorities... (In reply to Bernardo Gonzalez Kriegel from comment #2)

If the field/subfield is missing, adding it as 'hidden' does not affect present fw. And you can always preserve liblibrarian/libopac as they are... except when they change, as in last update (Bug 12435, bib 347$f).

As in other updates I proposed, I never try to update existing fw, for the problems that you commented.

It's a tough one. I'm thinking that there should be some mechanism for updating an existing framework. It might need to be triggered by library staff, so that they understand what is being updated, but there should be something there. Otherwise, if you're on Koha for a long time, you'll get stale frameworks. Since Koha used outdated frameworks for a long time, it's possible (re: probable) that many Koha users are using very out of date frameworks that need to be updated in a user-friendly way.

And there is a bigger problem: translations

I haven't done anything with translations, but I'm in favour of anything that makes translations easier.

...

We could actually just store this MARC-specific data in the YAML file, and then build the Default and other Custom frameworks based on a Perl script, which would probably be easier to maintain than the SQL that we maintain now.

If it's agreed to store fw in other format, I propose we use XML (example attached). And the reason is translations.

I'm happy to go with XML. I suppose my only reservations are about the brittleness of XML and processing time involved in parsing a huge XML file. I suppose the brittleness shouldn't be a factor...we could validate the XML before committing it and parsing errors are very loud so there probably is no danger there. As for processing time...it's not like installing or updating frameworks would be a frequent operation so lengthy processing time might not matter (if it is even a thing).

We need a way to ease the translation of all configuration and sample data we have as SQL. I'm working intermittently on this, now I'm stuck with sample notices, but have worked out most sql files as XML and a script to build sql from them. And unless we propose another way, we need translated sql files at web install time. It's not practical that each language other than english have a lang dir commited to git.

Now that I think about it... are "liblibrarian" and "libopac" the only fields that need to be translated?

Talking again about frameworks, Bug 10963 is waiting for a review :)

It looks good! I'll have to look at it soon. -- More thoughts that I'm having... marc_tag_structure: 1) MARC: tagfield, liblibrarian, libopac, repeatable 2) Koha: mandatory, authorised_value, frameworkcode marc_subfield_structure: 1) MARC: tagfield, tagsubfield, liblibrarian, libopac, repeatable, maxlength 2) Koha: mandatory, kohafield, tab, authorised_value, authtypecode, value_builder, isurl, hidden, frameworkcode, seealso, link, defaultvalue I'm wondering if it makes sense to maintain 2 separate sources of data. The first XML file would be the data that comes from the actual MARC specification (denoted by the "MARC:" tag above. The second XML file would be what Koha should do with the incoming data. -- Oooh. Brain wave...we could use XSLT! We'd have a XML file containing the MARC specification data. Then we pass that through an XSLT which adds the relevant "Koha" data. Think about it...in most cases those "Koha" fields are "null" or "". The most important fields are probably "tab" and "hidden". If we use defaults for most cases, but define exceptions in the XSLT on a field/subfield and framework basis...that would be much easier to maintain! I don't know if that paragraph makes sense to anyone else...but I like this idea. It's easier to maintain exceptions rather than a whole comprehensive list, which can be dynamically generated. Bernardo, if you could post an up-to-date XML file with that MARC data I mentioned above, I would be willing to take a look at creating a XSLT that adds the relevant Koha data and produces a final "Koha MARC framework" XML document in the end.

From there...we could use scripts to transform XML to SQL or to parse the XML and do the database updates via Perl.

Anyway...just some thoughts for now. Have to run! -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #5 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- (In reply to David Cook from comment #3)

Sorry for my delayed response! Competing priorities... No problem :) This task will take some time

I'm happy to go with XML. I suppose my only reservations are about the brittleness of XML and processing time involved in parsing a huge XML file. I suppose the brittleness shouldn't be a factor...we could validate the XML before committing it and parsing errors are very loud so there probably is no danger there. As for processing time...it's not like installing or updating frameworks would be a frequent operation so lengthy processing time might not matter (if it is even a thing).

I used XML::Simple to load and build sql files from the attached examples. It does not take to much time, and anyway is something to use at install time or when updating frameworks, not frequent tasks :)

Now that I think about it... are "liblibrarian" and "libopac" the only fields that need to be translated?

Yes, and in EN and ES they are the same text. I wonder if there are libraries using different labels for opac/staff In french UNIMARC frameworks I found that libopac is shortened or empty in most cases.

marc_tag_structure: 1) MARC: tagfield, liblibrarian, libopac, repeatable 2) Koha: mandatory, authorised_value, frameworkcode

In this case authorised_value... is not used, or can't be used. FW editor allows to set this to an authorised value, supposedly to list allowed values for indicators, but this is not used on editor (there is another bug that mentions this)

marc_subfield_structure: 1) MARC: tagfield, tagsubfield, liblibrarian, libopac, repeatable, maxlength 2) Koha: mandatory, kohafield, tab, authorised_value, authtypecode, value_builder, isurl, hidden, frameworkcode, seealso, link, defaultvalue

In this case seealso is not used anymore (or that is implied by http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6007#c3)

Oooh. Brain wave...we could use XSLT! We'd have a XML file containing the MARC specification data. Then we pass that through an XSLT which adds the relevant "Koha" data.

Think about it...in most cases those "Koha" fields are "null" or "". The most important fields are probably "tab" and "hidden". If we use defaults for most cases, but define exceptions in the XSLT on a field/subfield and framework basis...that would be much easier to maintain!

I don't know if that paragraph makes sense to anyone else...but I like this idea. It's easier to maintain exceptions rather than a whole comprehensive list, which can be dynamically generated.

Bernardo, if you could post an up-to-date XML file with that MARC data I mentioned above, I would be willing to take a look at creating a XSLT that adds the relevant Koha data and produces a final "Koha MARC framework" XML document in the end.

On the attached file there are XMLs for tags and subtags, with/without Koha proper fields. They are at least well formed (xmllint --noout file.xml) There's another thing to take into account: when MySQL 5.6 arrives to new debian/ubuntu versions, we'll have a problem. 5.6 client complains a lot to be used on CLI with password as argument, so Koha installer fails miserably because it uses 'mysql' command to load mandatory/optional values in database. Another reason to find other way. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- (In reply to Bernardo Gonzalez Kriegel from comment #6)

Created attachment 29869 [details] Loader for XML files

This new attach has a variant of XML files, with one element to represent data types, and a simple script to load the values into Koha database.

I tried $dbh->do() for each value, or a prepare() and then execute(), but is too slow. It builds a single string with all values to insert, similar as the SQL files for frameworks.

In my tests the script takes a little more than three times as long as mysql client to load default framework (tags/subtags), but is a difference between ~0.7s and ~2.5s, perhaps acceptable...

As you said before, the speed shouldn't matter too much if it's only being run during initial install and then during rare subsequent updates. I've been playing today with your XML files, and have been working on a script for updating existing frameworks using Koha::Database/DBIx::Class. The "find_or_new" method is quite useful in that case, as it'll insert new rows and it'll give you a chance to do further actions for rows that already exist (using the primary key as a reference point). At the moment, I have it set up to add missing fields/subfields, and update the "repeatable" column for existing fields if the XML values are different. I'm going to add an option for updating "libopac" and "liblibrarian" as well. I figure some libraries use custom text so they might not necessarily want to override it according to the MARC specification. Anyway... -- I can put together a demo DBIC loader for the default framework XML, but I'm wondering now how we want to handle installing the other sample frameworks. I took a look at bug 10963 and it looks really good. I like the idea you're going with. I'm thinking maybe something like this: http://search.cpan.org/~ribasushi/DBIx-Class-0.08270/lib/DBIx/Class/ResultSe... my $framework = $schema->resultset('MarcTagStructure')->search({ frameworkcode => '', }); then... $framework->create({ frameworkcode => 'SR' }); $framework->create({ frameworkcode => 'BKS' }); Then probably some updates to change the hidden values. I'm not sure where we would want to store the data about which subfields to hide in which framework though. I suppose we could hard-code it into the script, but it might be nicer to keep that information in a config file. Not 100% sure what the best way to update the hidden column will be...but I think that might be the way to do it. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 Jacek Ablewicz <abl@biblos.pk.edu.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |abl@biblos.pk.edu.pl --- Comment #9 from Jacek Ablewicz <abl@biblos.pk.edu.pl> --- (In reply to David Cook from comment #8)

If a field doesn't appear in the Editor, and it gets re-saved...any fields not appearing in the Editor will be destroyed.

Does it really work that way for biblos MARC frameworks?? If so, that would be a _major_ disaster IMO. I was under the impression that if the certain subfields are set up as "hidden in editor", they still get saved (not getting destroyed on re-save). Also, at least in the manner of speaking, they do appear in editor ;), their contents are just not visible and not editable (AFAIRC authority editor uses "display: none" for that purpose, I'm not sure about biblio editor).

I figure...if it doesn't appear in the Editor, it probably _doesn't_ need to appear anywhere else.

There are some scenarios when we e.g. may want some fields/subfields to be: preserved in the MARC record (control numbers, some linking data for records imported from union catalogues etc.), visible in staff client, but not directly editable by the librarians. Possibility to define (perhaps even greatly) simplified frameworks, intended for some specific editing tasks (= with a lot of existing record fields/subfields hidden on edit) might often come in handy as well. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- (In reply to Bernardo Gonzalez Kriegel from comment #10)

I created a new fw, set hidden = -6 for all subtags. A new record using this fw will only show those tags/subtags that are mandatory. Editing an existing record will show any tag/subtag that have a value, even hidden ones, and on saving no information is loose.

Ah, my mistake! That's good! I'm glad that I was wrong! It appears that this is also the case when using Z39.50. I think the problem I may have encountered in the past was losing data when importing a record that contained fields/subfields that didn't exist in that framework. *checks with an example* Aha! Yes! That's what it is! It makes sense...if the field/subfield doesn't exist in the framework, it has no way of including it in the record. I suppose this might be both a pro and a con... When copy-cataloguing, you might only want to import a certain subset of fields, so you'd want to ignore fields not defined in that framework. On the other hand, if you're cataloguing RDA records and you're not familiar with the extra fields and subfields needed for RDA, you'll lose data that you might want to keep when importing. That said, I think the onus needs to be on librarians to properly configure their frameworks for that. I don't know if this distinction is necessarily apparent in Koha. I imagine it's possible that a librarian could create a new framework with a subset of fields, and lose data when importing records because they forgot a field/subfield. But...you could argue the onus is still on the person configuring the framework. If they didn't include the field/subfield, they have no one but themselves to blame for lost data.

I think that default and sample (or simple) frameworks needs only 3 hidden values: visible (0), editable but not visible (2), and hidden (5) [visibility on opac/staff]. (5) could be used for obsoleted/deprecated tags/subtags (e.g. 440), but again you can't hide something in the editor with a value

(-6) is the value I was told to use when adding new tags/subtags, they do not display on editor (for new) and produce no functional change, but imported record can show values [opac/staff].

That sounds reasonable to me.

Whichever we use, my thinking is that we need to use a few hidden values as possible, and default FW needs to show as many tags as possible

Agreed.

Then sample FW could be built by copying default, and _hiding_ (editor) all but a selected subset of tags. Which subset... I don't care, but it need a certain rationale behind. Not an universal agreement though.

That also makes sense. I don't know which subset either. While there are cataloguing standards, I find practice is often highly localized, so they'll need to be re-configured by end-users anyway. It might be useful to get input from cataloguers that work with a range of materials for this one. I mostly catalogued books, so I would say Leader,005,008,100(a),245(a,b,c),260(a,b,c) (maybe also 264(a,b,c) as it's used in RDA),300(a),336(a,2),337(a,2),338(a,2),490(a,v,x),500(a),505(a,r,t),520(a,b),700(a,t,d,f). I don't have a lot of experience with serials but it should probably include 773(a,t,d,g,w). Other frameworks would probably benefit from cataloguers with more specialized knowledge and experience.

But if they have all tags/subs then no information will be loose. That's what we need to ensure

Agreed. In that case, what do you think about the Fast Add Framework? It only includes a small collection of fields/subfields. Wouldn't it make more sense to include all the fields, and only show the subset that it currently shows? While I'm somewhat skeptical of the utility of many of the sample frameworks, I think the Fast Add Framework is quite useful. Admittedly, I don't know if it needs to have all the fields, as I doubt one would ever really be importing records into it. I suppose it would be good to have all the fields in case you accidentally switched to the Fast Add Framework and saved a record. That would lose data... -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- Created attachment 29961 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=29961&action=edit DBIC-based framework updater Here is some more complex code that I've devised for updating frameworks. You give it an arrayref of hashrefs, a DBIC resultset, and an arrayref of framework codes. The script will check to see if a row matching the hashref (using the primary key specified in the resultset) exists for each framework. If it exists, it runs an optional "update" callback. If it doesn't exist, it runs an optional "insert" callback (or just inserts it into the database if no callback has been specified). Prior to the database search, the "canon_callback" can be used to modify the hashref for each frameworkcode. In my use case, if I'm loading marc_subfield_structure, I want to change the hidden value to -6 as I'm updating existing frameworks. Of course, if we were loading marc_tag_structure or were installing a framework for the first time, we wouldn't want to do that. I also use the insert/update callbacks to build messages that I sent to the template UI so that the user knows what has been updated. It's not very fast, but DBIC itself says "DBIx::Class is not built for speed, it's built for convenience and ease of use". It certainly is convenient and easy to use :). I have debated the utility of this class a bit. I figure it could be handy when we're installing the sample frameworks, as we could use the "canon_callback" to change the "hidden" value on a per framework and subfield basis. We wouldn't even need the insert/update callbacks. [Note: The DBIx::Class docs do mention that using find_or_create (and thus find_or_new or update_or_create) is subject to a race condition and that it should be run within a transaction. Mind you, lots of what we do in Koha could be subject to a race condition, so I don't know how important it is to get this right at the moment. In any case, there seems to be a few ways of using transactions with DBIC (http://search.cpan.org/dist/DBIx-Class/lib/DBIx/Class/Manual/Cookbook.pod#TR...). It's not really a big deal when initially installing frameworks, as there shouldn't be anyone to race against, but worth mentioning for updates and in general.] -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- tl;dr The top priority should be making sure that no unwanted data loss occurs. One solution might be to make sure that incoming fields/subfields are saved, even if they don't appear in a framework. (I suppose this solution has flaws in terms of determining labels and repeatability... I suppose you could say "unknown field/subfield" and default to being repeatable in the editor.) (For cataloguers who count on the workflow of fields/subfields not being saved if they're not in the framework, we could have a "strict" flag at the framework level which could control whether or not "unknown subfields/fields" are saved.) This solution would actually negate some of the need to update frameworks. Updates to labels and whether it's "repeatable" would be the only thing needed really, as the frameworks would be acting more as templates than as strict frameworks. Thoughts? -- You are receiving this mail because: You are watching all bug changes.