 |
 |
Hi folks, An update - thanks to a few folks, I've found out that there were some issues with process management near the end of the script, specifically between the csv_splitter.split call and the chunk of email code. For now, I decided to divide and conquer - I divided the main script into three separate files (parsing the MARC files into the big CSV file, splitting the CSV file, and emailing the notification) and created a script that calls each script in order. That seems to work for now :cP Cheers, and thanks again for all who chimed in, Becky On Wed, Jul 1, 2015 at 8:47 AM, Becky Yoose <[log in to unmask]> wrote: > Hi folks, > > I'm currently working on a script that would automatically generate CSV > files for my staff to upload into Serials Solutions to update our local > holdings. There is one part of the script that seems to keep failing. The > main script [1] currently creates one big CSV file which according to my > staff will create problems with the uploading process in Serials Solutions. > The current path I am on is to split this big CSV file into smaller files > for easier uploading. I found a small csv splitter script and modified it > to include quoting=csv.QUOTE_MINIMAL in the csv.writer call. [2] Created a > module from said code. So far, so good. > > When I call the csv_splitter.split in my main script, the function splits > the main CSV file into smaller files (default 10000 rows per file); > however, the last split file is missing at least 15-25 rows. I tested > different CSV files with the function and have seen the same issue of the > split file ending mid-row. I've uploaded the main CSV file and the last > output file from the function [3] - you will see that in line 9987 of the > output_5.csv file ends in the middle of the title. Again, I tested > different CSV files with this script and it consistently leaves out the > last few rows from the master CSV file. > > Am I missing something in the csv_splitter code that would cause an abrupt > end of writing rows to the output files? > > [1] Testing environment: Ubuntu 14.04.2 LTS, Python 2.7 - > https://gist.github.com/yo-bj/7224738839ff73837da6 > [2] https://gist.github.com/yo-bj/dea88123600d9a74b8ec > [3] > https://www.dropbox.com/sh/su81bcw7aiajepw/AAC_GjzAH7s1BbUwkF-Kf93Ha?dl=0 > > Thanks, > Becky > > ------------------------------------------------------- > Becky Yoose > Discovery and Integrated Systems Librarian > Grinnell College >