Eric-

I don't know the original source of those MARC files, but I've worked with files from an III system where diacritics had to be entered as character code escapes like "Muse{226}e du Louvre" (where 226 is the ANSEL code for a combining acute accent).  So if somebody made a typo and entered something like "Muse{22}6e du Louvre" instead, you'd get some bogus invalid character.  I was working with MARCXML files in Java, so I wrote a FilterReader class that removed any characters that were invalid in UTF-8 XML.  I assume you could do something similar in Perl (probably with a fancy one-line regex).

-Esme
--
Esme Cowles <[log in to unmask]>

"We've all heard that a million monkeys banging on a million typewriters
 will eventually reproduce the works of Shakespeare. Now, thanks to the
 Internet, we know this is not true." -- Robert Wilensky

On Oct 7, 2010, at 6:51 AM, Eric Lease Morgan wrote:

> How do I trap for unwanted (bogus) characters in MARC records?
> 
> I have a set of Internet Archive identifiers, and have written the followoing Perl loop to get the MARC records associated with each one:
> 
>  # process each identifier
>  my $ua = LWP::UserAgent->new( agent => AGENT );
>  while ( <DATA> ) {
> 
>    # get the identifier
>    chop;
>    my $identifier = $_;
>    print $identifier, "\n";
> 
>    # get its corresponding MARC record
>    my $response = $ua->get( ROOT . "$identifier/$identifier" . "_meta.mrc" );
>    if ( ! $response->is_success ) {
> 
>      warn $response->status_line;
>      next;
> 
>    }
> 
>    # save it
>    open MARC, " > $identifier.mrc" or die "Can't open $identifier.mrc: $!\n";
>    binmode MARC, ":utf8";
>    print MARC $response->content;
>    close MARC;
> 
>  }
> 
> I then use the venerable marcdump to see the fruits of my labors: marcdump *.mrc. Unfortunately, marcdump returns the following error against (at least) one of my files:
> 
>  bienfaitsducatho00pina.mrc
>  utf8 "\xC3" does not map to Unicode at /System/Library/
>  Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162.
> 
> What is going on here? Am I saving my files incorrectly? Is the original MARC data inherintly incorrect? Is there some way I can fix the MARC record in question?
> 
> -- 
> Eric Lease Morgan