cron job for loading pictures or better option
I have 53,000 students and a windows server in another building that I have a fileshare to. What would be the recommended way to copy those files over and load them into Koha? I have tried scp * but the problem is that it only copies what 2000 at a time and then dies. I'm a librarian looking for an easy solution without lots of programming. So here was my thought: build a cron first make the fileshare connection then issue several scp statements to copy the files over to the local server then issue the command to batch load them I have a command line utility that someone gave me to batch load pictures. I'll find someone to send it to for testing with 3.2. Next task... Thoughts? Comments - is there an easier way? manually I have copied the files over and run this command to load them I believe the utility is assuming the files are in the same directory: ./loadimages.sh - which issues perl loadimages.pl -i IDLINK.txt -l image.log from the loadimages.pl file - my(%count,$input_file,$error_file,$log_file,$sepchar,$imagedirectory); $imagedirectory = ""; -- View this message in context: http://old.nabble.com/cron-job-for-loading-pictures-or-better-option-tp27243... Sent from the Koha - Dev mailing list archive at Nabble.com.
I also see this line in the .pl file ##example -- > perl loadimages.pl -i IDLINK.txt -l image.log -d "/home/pisd/dataload/pics/" so if there was a way to link the the sharefile and not actually scp the files over that would be way cool. David Schuster wrote:
I have 53,000 students and a windows server in another building that I have a fileshare to. What would be the recommended way to copy those files over and load them into Koha?
I have tried scp * but the problem is that it only copies what 2000 at a time and then dies.
I'm a librarian looking for an easy solution without lots of programming.
So here was my thought:
build a cron first make the fileshare connection then issue several scp statements to copy the files over to the local server then issue the command to batch load them
I have a command line utility that someone gave me to batch load pictures. I'll find someone to send it to for testing with 3.2. Next task...
Thoughts? Comments - is there an easier way?
manually I have copied the files over and run this command to load them I believe the utility is assuming the files are in the same directory:
./loadimages.sh - which issues perl loadimages.pl -i IDLINK.txt -l image.log
from the loadimages.pl file -
my(%count,$input_file,$error_file,$log_file,$sepchar,$imagedirectory); $imagedirectory = "";
-- View this message in context: http://old.nabble.com/cron-job-for-loading-pictures-or-better-option-tp27243... Sent from the Koha - Dev mailing list archive at Nabble.com.
Hi David, I think Rsync is your answer. You can use it on both boxes. It will keep the directories in sync without moving all 53K files each time. Win32: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp *nix: http://samba.anu.edu.au/rsync/ Kind Regards, Chris On Wed, Jan 20, 2010 at 10:53 AM, David Schuster <dschust1@tx.rr.com> wrote:
I also see this line in the .pl file
##example -- > perl loadimages.pl -i IDLINK.txt -l image.log -d "/home/pisd/dataload/pics/"
so if there was a way to link the the sharefile and not actually scp the files over that would be way cool.
David Schuster wrote:
I have 53,000 students and a windows server in another building that I have a fileshare to. What would be the recommended way to copy those files over and load them into Koha?
I have tried scp * but the problem is that it only copies what 2000 at a time and then dies.
I'm a librarian looking for an easy solution without lots of programming.
So here was my thought:
build a cron first make the fileshare connection then issue several scp statements to copy the files over to the local server then issue the command to batch load them
I have a command line utility that someone gave me to batch load
pictures.
I'll find someone to send it to for testing with 3.2. Next task...
Thoughts? Comments - is there an easier way?
manually I have copied the files over and run this command to load them I believe the utility is assuming the files are in the same directory:
./loadimages.sh - which issues perl loadimages.pl -i IDLINK.txt -l image.log
from the loadimages.pl file -
my(%count,$input_file,$error_file,$log_file,$sepchar,$imagedirectory); $imagedirectory = "";
-- View this message in context: http://old.nabble.com/cron-job-for-loading-pictures-or-better-option-tp27243... Sent from the Koha - Dev mailing list archive at Nabble.com.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
It's possible in linux to connect to a windows share. A command like the following would do the trick. mount -t smbfs //your_server.full.path/share /mnt/point -o workgroup=your_workgroup,username=a_username,password=read_only_password Then you can load the images from /mnt/point. On Wed, 2010-01-20 at 07:53 -0800, David Schuster wrote:
I also see this line in the .pl file
##example -- > perl loadimages.pl -i IDLINK.txt -l image.log -d "/home/pisd/dataload/pics/"
so if there was a way to link the the sharefile and not actually scp the files over that would be way cool.
David Schuster wrote:
I have 53,000 students and a windows server in another building that I have a fileshare to. What would be the recommended way to copy those files over and load them into Koha?
I have tried scp * but the problem is that it only copies what 2000 at a time and then dies.
I'm a librarian looking for an easy solution without lots of programming.
So here was my thought:
build a cron first make the fileshare connection then issue several scp statements to copy the files over to the local server then issue the command to batch load them
I have a command line utility that someone gave me to batch load pictures. I'll find someone to send it to for testing with 3.2. Next task...
Thoughts? Comments - is there an easier way?
manually I have copied the files over and run this command to load them I believe the utility is assuming the files are in the same directory:
./loadimages.sh - which issues perl loadimages.pl -i IDLINK.txt -l image.log
from the loadimages.pl file -
my(%count,$input_file,$error_file,$log_file,$sepchar,$imagedirectory); $imagedirectory = "";
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
Mike you are the man! I have figured out the correct perl command to load them from the remote server without transfering the files to my server... OH THIS IS GREAT! I'll see if it works with 3.2 and maybe we can get it included for the rest of the world that has thousands of pictures to load! I'm going to set this up as a cron job to load once a week... Way cool. Now if I could just get the cron job setup to load borrowers daily from a command line script... anyone anyone....??? David Schuster Mike Hafen-2 wrote:
It's possible in linux to connect to a windows share. A command like the following would do the trick.
mount -t smbfs //your_server.full.path/share /mnt/point -o workgroup=your_workgroup,username=a_username,password=read_only_password
Then you can load the images from /mnt/point.
On Wed, 2010-01-20 at 07:53 -0800, David Schuster wrote:
I also see this line in the .pl file
##example -- > perl loadimages.pl -i IDLINK.txt -l image.log -d "/home/pisd/dataload/pics/"
so if there was a way to link the the sharefile and not actually scp the files over that would be way cool.
David Schuster wrote:
I have 53,000 students and a windows server in another building that I have a fileshare to. What would be the recommended way to copy those files over and load them into Koha?
I have tried scp * but the problem is that it only copies what 2000 at
time and then dies.
I'm a librarian looking for an easy solution without lots of
a programming.
So here was my thought:
build a cron first make the fileshare connection then issue several scp statements to copy the files over to the local server then issue the command to batch load them
I have a command line utility that someone gave me to batch load
pictures.
I'll find someone to send it to for testing with 3.2. Next task...
Thoughts? Comments - is there an easier way?
manually I have copied the files over and run this command to load them I believe the utility is assuming the files are in the same directory:
./loadimages.sh - which issues perl loadimages.pl -i IDLINK.txt -l image.log
from the loadimages.pl file -
my(%count,$input_file,$error_file,$log_file,$sepchar,$imagedirectory); $imagedirectory = "";
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- View this message in context: http://old.nabble.com/cron-job-for-loading-pictures-or-better-option-tp27243... Sent from the Koha - Dev mailing list archive at Nabble.com.
David, If you want to write up a little FAQ with the perl command you're using, I'll put it in the manual for all to benefit from while we wait to get the code into Koha. Nicole On Fri, Jan 22, 2010 at 10:31 AM, David Schuster <dschust1@tx.rr.com> wrote:
Mike you are the man! I have figured out the correct perl command to load them from the remote server without transfering the files to my server... OH THIS IS GREAT!
I'll see if it works with 3.2 and maybe we can get it included for the rest of the world that has thousands of pictures to load!
I'm going to set this up as a cron job to load once a week... Way cool. Now if I could just get the cron job setup to load borrowers daily from a command line script... anyone anyone....???
David Schuster
Mike Hafen-2 wrote:
It's possible in linux to connect to a windows share. A command like the following would do the trick.
mount -t smbfs //your_server.full.path/share /mnt/point -o workgroup=your_workgroup,username=a_username,password=read_only_password
Then you can load the images from /mnt/point.
On Wed, 2010-01-20 at 07:53 -0800, David Schuster wrote:
I also see this line in the .pl file
##example -- > perl loadimages.pl -i IDLINK.txt -l image.log -d "/home/pisd/dataload/pics/"
so if there was a way to link the the sharefile and not actually scp the files over that would be way cool.
David Schuster wrote:
I have 53,000 students and a windows server in another building that I have a fileshare to. What would be the recommended way to copy those files over and load them into Koha?
I have tried scp * but the problem is that it only copies what 2000 at
time and then dies.
I'm a librarian looking for an easy solution without lots of
a programming.
So here was my thought:
build a cron first make the fileshare connection then issue several scp statements to copy the files over to the local server then issue the command to batch load them
I have a command line utility that someone gave me to batch load
pictures.
I'll find someone to send it to for testing with 3.2. Next task...
Thoughts? Comments - is there an easier way?
manually I have copied the files over and run this command to load them I believe the utility is assuming the files are in the same directory:
./loadimages.sh - which issues perl loadimages.pl -i IDLINK.txt -l image.log
from the loadimages.pl file -
my(%count,$input_file,$error_file,$log_file,$sepchar,$imagedirectory); $imagedirectory = "";
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- View this message in context: http://old.nabble.com/cron-job-for-loading-pictures-or-better-option-tp27243... Sent from the Koha - Dev mailing list archive at Nabble.com.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
I've actually got a feature in my personal code base to manually synchronize patrons with another database, but that's probably not what you're looking for here. If that would work for you then you should be able to pull the patch from my gitweb server. See my signature for the address. It is the first patch ( at the bottom of the list of patches authored by me ). On Fri, 2010-01-22 at 07:31 -0800, David Schuster wrote:
I'm going to set this up as a cron job to load once a week... Way cool. Now if I could just get the cron job setup to load borrowers daily from a command line script... anyone anyone....???
David Schuster
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
participants (4)
-
Chris Nighswonger -
David Schuster -
Michael Hafen -
Nicole Engard