[Bug 11368] New: Add script to import Lexile scores
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Bug ID: 11368 Summary: Add script to import Lexile scores Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: gmcharlt@gmail.com Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Koha needs a script to automate the importing of Lexile score data for titles that have available scores but are not currently in the title's record. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |kyle@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #1 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 23379 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23379&action=edit Very small example Lexile CSV file -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23379|Very small example Lexile |LexileTitlesTruncated.txt description|CSV file | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #2 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 23380 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23380&action=edit Bug 11368 - Add script to import Lexile scores Koha needs a script to automate the importing of Lexile score data for titles that have available scores but are not currently in the title's record. This script will take a CSV file of Lexile scores, and locate any matching records in the Koha database ( by ISBN ). If the record already has a score, it will be updated. If not, the Lexile score field will be created. Test Plan: 1) Apply this patch 2) Catalog a record for each of the following ISBNs: 0789170191 9780673779410 3) Download the file LexileTitlesTruncated.txt attached to this bug report 4) Run the script from the command line: ./misc/migraction_tools/import_lexile.pl -v --file /path/to/LexileTitlesTruncated.txt 5) View those records in Koha 6) Note those records now have valid Lexile scores 7) Edit the Lexile score ( 521$a ) and change the value to something else 8) Repeat step 4 9) Note the original Lexile score has been restored -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |10500 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Nicole C. Engard <nengard@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nengard@gmail.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23380|0 |1 is obsolete| | --- Comment #3 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 28092 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=28092&action=edit Bug 11368 - Add script to import Lexile scores Koha needs a script to automate the importing of Lexile score data for titles that have available scores but are not currently in the title's record. This script will take a CSV file of Lexile scores, and locate any matching records in the Koha database ( by ISBN ). If the record already has a score, it will be updated. If not, the Lexile score field will be created. Test Plan: 1) Apply this patch 2) Catalog a record for each of the following ISBNs: 0789170191 9780673779410 3) Download the file LexileTitlesTruncated.txt attached to this bug report 4) Run the script from the command line: ./misc/migraction_tools/import_lexile.pl -v --file /path/to/LexileTitlesTruncated.txt 5) View those records in Koha 6) Note those records now have valid Lexile scores 7) Edit the Lexile score ( 521$a ) and change the value to something else 8) Repeat step 4 9) Note the original Lexile score has been restored -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gmcharlt@gmail.com --- Comment #4 from Galen Charlton <gmcharlt@gmail.com> --- Out of curiosity, who provides Lexile scores in that format? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=7377 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Mirko Tietgen <mirko@abunchofthings.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #28092|0 |1 is obsolete| | --- Comment #5 from Mirko Tietgen <mirko@abunchofthings.net> --- Created attachment 36438 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=36438&action=edit Bug 11368 - Add script to import Lexile scores Koha needs a script to automate the importing of Lexile score data for titles that have available scores but are not currently in the title's record. This script will take a CSV file of Lexile scores, and locate any matching records in the Koha database ( by ISBN ). If the record already has a score, it will be updated. If not, the Lexile score field will be created. Test Plan: 1) Apply this patch 2) Catalog a record for each of the following ISBNs: 0789170191 9780673779410 3) Download the file LexileTitlesTruncated.txt attached to this bug report 4) Run the script from the command line: ./misc/migraction_tools/import_lexile.pl -v --file /path/to/LexileTitlesTruncated.txt 5) View those records in Koha 6) Note those records now have valid Lexile scores 7) Edit the Lexile score ( 521$a ) and change the value to something else 8) Repeat step 4 9) Note the original Lexile score has been restored Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Mirko Tietgen <mirko@abunchofthings.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |mirko@abunchofthings.net -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de Status|Signed Off |Failed QA --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Kyle, 0) The QA script fails: FAIL misc/migration_tools/import_lexile.pl FAIL critic I/O layer ":utf8" used at line 39, column 1. Use ":encoding(UTF-8)" to get strict validation. FAIL forbidden patterns forbidden pattern: Koha is now under the GPLv3 license (line 9) OK pod OK valid 1) Is copyright 2013 correct? 2) I am not sure what one of the options means from the description, could you clarify? + --source-value Defines the value to put stored in the "Source" subfield ( default: "Lexile" ) 3) The documentation for --file says: "acquired from Lexile.com", but the actual file format is not documented. I think maybe we need to decide here if this is a vendor specific or a general purpose script. If it's vendor specific, we might want to reflect that more clearly in the file name and description. If it's general purpose, we should include notes about the file format to make this easier to reuse. 4) We didn't really finish our discussions about the standard options for CLI scripts (http://wiki.koha-community.org/wiki/Coding_Guidelines#Command-line_argument_...), but maybe this should have a -c for confirming the action as it's going to change data? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #7 from Kyle M Hall <kyle@bywatersolutions.com> ---
0) The QA script fails:
FAIL misc/migration_tools/import_lexile.pl FAIL critic I/O layer ":utf8" used at line 39, column 1. Use ":encoding(UTF-8)" to get strict validation. FAIL forbidden patterns forbidden pattern: Koha is now under the GPLv3 license (line 9) OK pod OK valid
Will fix!
1) Is copyright 2013 correct?
Will fix!
2) I am not sure what one of the options means from the description, could you clarify? + --source-value Defines the value to put stored in the "Source" subfield ( default: "Lexile" )
Please take a look at http://www.loc.gov/marc/bibliographic/bd521.html Basically, 521 fields can store all types of target audience data. 521$a stores the 'rating' while 521$b stores the type of the rating. In this case, the rating is a Lexile score.
3) The documentation for --file says: "acquired from Lexile.com", but the actual file format is not documented. I think maybe we need to decide here if this is a vendor specific or a general purpose script. If it's vendor specific, we might want to reflect that more clearly in the file name and description. If it's general purpose, we should include notes about the file format to make this easier to reuse.
I'll add the file format to the help. This is meant specifically for importing Lexile.com files ( as they own and control Lexile rankings ) but this script is general purpose enough to import any CSV file that matches the schema.
4) We didn't really finish our discussions about the standard options for CLI scripts (http://wiki.koha-community.org/wiki/Coding_Guidelines#Command- line_argument_conventions), but maybe this should have a -c for confirming the action as it's going to change data?
I'll add the -c and -t options. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #36438|0 |1 is obsolete| | --- Comment #8 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 39182 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=39182&action=edit Bug 11368 - Add script to import Lexile scores Koha needs a script to automate the importing of Lexile score data for titles that have available scores but are not currently in the title's record. This script will take a CSV file of Lexile scores, and locate any matching records in the Koha database ( by ISBN ). If the record already has a score, it will be updated. If not, the Lexile score field will be created. Test Plan: 1) Apply this patch 2) Catalog a record for each of the following ISBNs: 0789170191 9780673779410 3) Download the file LexileTitlesTruncated.txt attached to this bug report 4) Run the script from the command line: ./misc/migraction_tools/import_lexile.pl -v --file /path/to/LexileTitlesTruncated.txt 5) View those records in Koha 6) Note those records now have valid Lexile scores 7) Edit the Lexile score ( 521$a ) and change the value to something else 8) Repeat step 4 9) Note the original Lexile score has been restored Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #9 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 39183 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=39183&action=edit Bug 11368 [QA Followup] * Fix QA failures * Fix copyright * Add file format documentation * Add -c --confirm option * Add -t --test option * Add -h --help option -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #10 from Mirko Tietgen <mirko@abunchofthings.net> --- (In reply to Katrin Fischer from comment #6)
If it's vendor specific, we might want to reflect that more clearly in the file name and description. If it's general purpose, we should include notes about the file format to make this easier to reuse.
I thought about broadening the scope of the script when I signed off on the patch, checking with the table for Antolin (German reading platform with quizzes for school kids, run by a publisher). I don't think it's feasible to guess all possible fields and options that may occur in such lists, and given the non-standardized nature of these things, you never know if a format will change at some point. It is easily possible to tweak the script if you know what you need, so I'd say keeping it vendor specific and having more vendor specific scripts of this nature if needed makes sense. This patch is a good starting point for similar tasks. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #11 from Jonathan Druart <jonathan.druart@biblibre.com> --- Comment on attachment 39182 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=39182 Bug 11368 - Add script to import Lexile scores Review of attachment 39182: --> (http://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=11368&attachment=39182) ----------------------------------------------------------------- I get this warning: DBIx::Class::ResultSet::search_rs(): search( %condition ) is deprecated, use search( \%condition ) instead at misc/migration_tools/import_lexile.pl line 147 ::: misc/migration_tools/import_lexile.pl @@ +65,5 @@
+ 'source=s' => $subfield_source, + 'source-value=s' => $subfield_source_value, +); + +my $usage = << 'ENDUSAGE';
Should not we use Pod::Usage? @@ +82,5 @@
+ +ENDUSAGE + +unless ($file) { + say $usage;
Prefer to call pod2usage. @@ +88,5 @@
+} + +my $schema = Koha::Database->new()->schema(); + +my $csv = Text::CSV->new( { binary => 1, sep_char => "\t" } )
Shouldn't we use the pref "separator" and add an option to specify another one? @@ +121,5 @@
+ push( @isbns, $row->{$_} ); + eval { push( @isbns, GetVariationsOfISBN( $row->{$_} ) ) }; + } + } + @isbns = grep( $_, @isbns );
Not sure to undestand what this does :) @@ +142,5 @@
+ + if ($verbose) { + say "Found matching record! Biblionumber: $biblionumber"; + + if ( $verbose > 2 ) {
You should mention this in the POD. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |jonathan.druart@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Nicole C. Engard <nengard@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nicole@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #12 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 42603 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42603&action=edit Bug 11368 [QA Followup 2] -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Kyle M Hall <kyle.m.hall@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off CC| |kyle.m.hall@gmail.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #13 from Kyle M Hall <kyle.m.hall@gmail.com> --- (In reply to Jonathan Druart from comment #11)
Comment on attachment 39182 [details] [review] Bug 11368 - Add script to import Lexile scores
Review of attachment 39182 [details] [review]: -----------------------------------------------------------------
I get this warning: DBIx::Class::ResultSet::search_rs(): search( %condition ) is deprecated, use search( \%condition ) instead at misc/migration_tools/import_lexile.pl line 147
Fixed!
::: misc/migration_tools/import_lexile.pl @@ +65,5 @@
+ 'source=s' => $subfield_source, + 'source-value=s' => $subfield_source_value, +); + +my $usage = << 'ENDUSAGE';
Should not we use Pod::Usage?
@@ +82,5 @@
+ +ENDUSAGE + +unless ($file) { + say $usage;
Prefer to call pod2usage.
I don't think the developer guidelines require pod2usage. That being said, I'd welcome a followup!
@@ +88,5 @@
+} + +my $schema = Koha::Database->new()->schema(); + +my $csv = Text::CSV->new( { binary => 1, sep_char => "\t" } )
Shouldn't we use the pref "separator" and add an option to specify another one?
The file format is fixed with tabs as the separator, so we should definitely not use the syspref here.
@@ +121,5 @@
+ push( @isbns, $row->{$_} ); + eval { push( @isbns, GetVariationsOfISBN( $row->{$_} ) ) }; + } + } + @isbns = grep( $_, @isbns );
Not sure to undestand what this does :)
This line filters out any 'empty' isbns.
@@ +142,5 @@
+ + if ($verbose) { + say "Found matching record! Biblionumber: $biblionumber"; + + if ( $verbose > 2 ) {
You should mention this in the POD.
Done! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #39182|0 |1 is obsolete| | Attachment #39183|0 |1 is obsolete| | Attachment #42603|0 |1 is obsolete| | --- Comment #14 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42641 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42641&action=edit Bug 11368 - Add script to import Lexile scores Koha needs a script to automate the importing of Lexile score data for titles that have available scores but are not currently in the title's record. This script will take a CSV file of Lexile scores, and locate any matching records in the Koha database ( by ISBN ). If the record already has a score, it will be updated. If not, the Lexile score field will be created. Test Plan: 1) Apply this patch 2) Catalog a record for each of the following ISBNs: 0789170191 9780673779410 3) Download the file LexileTitlesTruncated.txt attached to this bug report 4) Run the script from the command line: ./misc/migraction_tools/import_lexile.pl -v --file /path/to/LexileTitlesTruncated.txt 5) View those records in Koha 6) Note those records now have valid Lexile scores 7) Edit the Lexile score ( 521$a ) and change the value to something else 8) Repeat step 4 9) Note the original Lexile score has been restored Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #15 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42642 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42642&action=edit Bug 11368 [QA Followup] * Fix QA failures * Fix copyright * Add file format documentation * Add -c --confirm option * Add -t --test option * Add -h --help option -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42643 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42643&action=edit Bug 11368 [QA Followup 2] Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com Status|Passed QA |Pushed to Master --- Comment #17 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Patches pushed to master. Thanks Kyle! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11368 Heather Braum <hbraum@nekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hbraum@nekls.org -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org