 |
 |
Hi again folks: Many thanks for everyone's replies this week! We figured it out here with y'all's help, & so I wanna share back what we learned. Like folks suggested, the problem was some combination of line breaks, commas & double quotes inside values. Maintaining the breaks/characters isn't important for us. So I used OpenRefine to "trim leading & trailing whitespace" & "collapse consecutive whitespace" at our very messy OCR transcript column to clean it up some. & then used find & replace at Excel to replace the breaks, commas & double quotes. & success! 👍👍 Thanks again for everyone's replies & personal messages & help! Y'all are great folks. 🙏🙏 Cheers all, Max Maxwell Gray https://maxgray20.com On Tue, Jan 17, 2023 at 10:49 AM Geoffrey Spear <[log in to unmask]> wrote: > Nitpick: RFC 7111 (the standard for CSV files in MIME, although, as noted > by others in this thread, CSV in practice might as well not have a standard > at all since different tools feel free to do whatever the heck they want) > says to always use Windows-style line endings in CSV files (CRLF), whatever > system you're on. > > I don't see any explicit line ending handling in the github repo you linked > to, but other things will likely expect this ending (including, by not > limited to, Python's csv library which also has some suggestions for usage > to fix things.) > > CSVImport also seems to raise an entirely different error message if it > thinks your file isn't UTF-8, but it doesn't hurt to verify in advance. > It's certainly one of the easier encodings to tell a bunch of bytes > definitely isn't using, but I didn't verify whether CSVImport is actually > doing that in a foolproof way... > > On Tue, Jan 17, 2023 at 10:08 AM Benjamin Armintor <[log in to unmask]> > wrote: > > > Sometimes in a situation like this it's useful to look at the source of > the > > message. In the event that this is true here... > > > > This error is raised by the CSV import module here: > > > > > https://github.com/omeka-s-modules/CSVImport/blob/30857ff5cbab31bb53713fc7c837b8c2c1247f6e/src/Source/AbstractSource.php#L89-L94 > > The checkNumberOfColumnsByRow function returning false (and triggering > the > > error) is here: > > > > > https://github.com/omeka-s-modules/CSVImport/blob/30857ff5cbab31bb53713fc7c837b8c2c1247f6e/src/Source/CsvFile.php#L126-L144 > > > > That function appears to verify two things: > > 1. The iterator over the CSV is not empty > > 2. The rows all have the same number of values as the header row had > > headers > > > > The iterator in question is from an SplFileObject ( > > https://www.php.net/manual/en/class.splfileobject.php). So either your > > uploaded file appears empty to the CSV reader, or some row has a number > of > > cells different from the header row. If I were you, before I started > > digging into escaped values and what not, I would: > > 1. Make sure I had the number of headers I intended > > 2. Make sure my file is UTF8 encoded, and make sure I was using > Unix-style > > line endings (a single newline character) > > 3. Make sure I didn't have empty lines (I notice that > > > > > https://github.com/omeka-s-modules/CSVImport/blob/30857ff5cbab31bb53713fc7c837b8c2c1247f6e/src/Source/CsvFile.php#L189-L190 > > sets the SKIP_EMPTY flag, but not DROP_NEW_LINE), including the end of > the > > file > > > > If you've already done these things, I apologize for being so > rudimentary - > > but it's always good to verify the basic assumptions before you dive into > > more elaborate data inspection. > > > > Good luck! > > Ben > > > > PS: I know this stuff is frustrating - it might be worth opening an issue > > on that CSVImport github repository to improve the error message! > > > > On Tue, Jan 17, 2023 at 9:15 AM Max <[log in to unmask]> wrote: > > > > > Hi again folks: > > > > > > Many thanks for everyone's replies yesterday evening! I retried fixing > > via > > > OpenRefine, & no success. & I'm using double quotes for CSV comma > > > enclosure. So I don't think commas inside values are the problem. (I've > > had > > > success in the past with multiple different CSV files that included > extra > > > commas inside values & used double quotes for CSV comma enclosure, & so > > > they weren't a problem.) RE: counting blanks in columns, Jackie Keith > > > recommended using the COUNTBLANK formula at Excel/Google Sheets, which > > was > > > easy to use, but still no success. (I didn't find any blanks in the > > data.) > > > Anyway, I wanted to update folks, & say thanks again, everyone! > > > > > > Cheers all, > > > Max > > > > > > Maxwell Gray > > > https://maxgray20.com > > > > > > > > > On Mon, Jan 16, 2023 at 8:59 PM Joe Hourclé <[log in to unmask]> > > wrote: > > > > > > > > On Jan 16, 2023, at 7:26 PM, Max <[log in to unmask]> wrote: > > > > > > > > > > Hi code4lib folks: > > > > > > > > > > Does anyone know a tool or hack to help fix a problem at a CSV > that's > > > > > causing a "The rows are not all the same number of columns." error > > when > > > > > trying to import the CSV at a web application? I'm trying to use > the > > > CSV > > > > > Import module < > > https://omeka.org/s/docs/user-manual/modules/csvimport/ > > > > > > > > at > > > > > Omeka S. I've had success in the past with different CSV files. But > > > some > > > > > kinda problem at the CSV I'm trying to import right now is causing > > this > > > > > error, & reviewing the CSV in Excel & as plain text (literally > > counting > > > > > commas to confirm rows are the same number of columns) isn't > helping. > > > > > > > > I’ve been known to do a find/replace on commas, then set the tab > width > > to > > > > something very large and then look for the rows that don’t line up. > > > > > > > > But CSV is tricky, as it’s more than just commas that are > significant, > > > you > > > > also have to consider quotations marks… which allow it so you can put > > > > commas or line returns within a string field. > > > > > > > > -Joe > > > > > > > > Sent from a mobile device with a crappy on screen keyboard and > > obnoxious > > > > "autocorrect" > > > > > > > > > >