 |
 |
On Jan 23, 2009, at 5:52 AM, Eric Lease Morgan wrote: > On 1/23/09 4:39 AM, "Brown, Alan" <[log in to unmask]> wrote: > >>> Does anybody here know the difference between MARC21 and USMARC? >>> >>> I am munging sets of MARC bibliographic data from a III catalog with >>> holdings data from the same. I am using MARC::Batch to read my bib' >>> data (with both strict and warnings turned off), insert 853 and 863 >>> fields, and writing the data using the as_usmarc method. >>> Therefore, I >>> think I am creating USMARC files. I can then use marcdump to... dump >>> the records. It returns 0 errors. >> >> Eric, This isn't an encoding thing is it? I know that a number of III >> catalogues still encode their diacritics using the MARC8 version of >> USMARC. We have changed ours to Unicode now, but we did have an >> issue of >> the catalogue outputting unicode records that weren't tagged as >> such in >> the leader and so couldn't be identified as proper MARC21 (current >> version of USMARC). III have solved this with their latest release. >> This >> issue had me scratching my head with a lot of my MARC::Record >> scripts, >> but generally they failed quite spectacularly. > > > Actually, I believe I am suffering from a number of different types of > errors in my MARC data: 1) encoding issues (MARC8 versus UTF-8), 2) > syntactical errors (lack of periods, invalid choices of indicators, > etc.), > 3) incorrect data types (strings entered into fields denoted for > integers, > etc.) Just about the only thing I haven't encountered are structural > errors > such as invalid leader, and this doesn't even take into account > possible > data entry errors (author is Franklin when Twain was entered). > > Yes, I do have an encoding issue. All of my incoming records are in > MARC8. > I'm not sure, but I think the Primo tool expects UTF-8. I can easily > update > the encoding bit (change leader position 09 from blank to a), but > this does > not change any actual encoding in the bibliographic section of my > data. > Consequently, after updating the encoding bit and looping through my > munged > data MARC::Record chokes on records with the following error where > UTF-8 is > denoted but include MARC8 characters: > > utf8 "\xE8" does not map to Unicode at > /usr/lib/perl5/5.8.8/i686-linux/Encode.pm line 166. > > Upon looking at the raw MARC see the the offending record includes > the word > Münich. What can I do to transform MARC8 data into UTF-8? What can I > do to > trap the error above, and skip these invalid records? We've had good luck with the yaz-marcdump utility that's included with the YAZ toolkit. We're using it to convert our exported Horizon records from MARC8 to UTF-8 before we import into AquaBrowser. The tool is easy to compile, blindingly fast, forgiving of common MARC errors, and changes the coding correctly. It's been serving us well. -Tod Tod Olson <[log in to unmask]> Systems Librarian University of Chicago Library