Eric, is this your source file?
http://ia341306.us.archive.org/1/items/bienfaitsducatho00pina/bienfaitsducatho00pina_meta.mrc
I have nothing really much to offer with regard to MARC.pm and its
ilk, but I thought it might help people track down your problem.
FWIW, yaz-marcdump spits out this on that record:
$ yaz-marcdump bienfait.mrc
00795cam a2200277 a 4500
001 1556719
003 CaOTULAS
005 19931129144435.0
008 780210s1842 fr fre d
035 $a (Sirsi) AZF-9578
040 $a NUC $c NUC $d otsm
049 $a otstm $b eng
050 04 $a BX946 $b .P5
055 3 $a BX946 $b .P55 1842
090 8 $a BX 946 .P55 1842 $b SMRS
100 10 $a Pinard, Clovis, $d d.1865.
245 10 $a Bienfaits du Catholicisme dans la société / $c par l'abbé P
(No separator at end of field length=71)
260 na $d .
(Separator but not at end of field length=26)
300 18 $2 .
(Separator but not at end of field length=11)
490 00 $p .
(Separator but not at end of field length=45)
(Bad indicator data. Skipping 2 bytes)
596 ?t $e nn
(No separator at end of field length=7)
610 ne
(Separator but not at end of field length=30)
948 xH $s tory.
(Separator but not at end of field length=27)
039 0/ $6 /199
(No separator at end of field length=9)
(Bad indicator data. Skipping 1 bytes)
093 0 $f mcsk
(Separator but not at end of field length=21)
926 12 $1 44434
(Separator but not at end of field length=48)
The diacritics definitely look pretty sketchy there.
In fact, I just tried this with every encoding in yaz-marcdump, and
the diacritics never properly converted to UTF-8.
They seem ok here:
http://ia341306.us.archive.org/1/items/bienfaitsducatho00pina/bienfaitsducatho00pina_marc.xml
though, so you might want to grab both binary marc and marcxml and
fall back to the latter in case of encoding errors.
-Ross.
On Thu, Oct 7, 2010 at 6:51 AM, Eric Lease Morgan <[log in to unmask]> wrote:
> How do I trap for unwanted (bogus) characters in MARC records?
>
> I have a set of Internet Archive identifiers, and have written the followoing Perl loop to get the MARC records associated with each one:
>
> # process each identifier
> my $ua = LWP::UserAgent->new( agent => AGENT );
> while ( <DATA> ) {
>
> # get the identifier
> chop;
> my $identifier = $_;
> print $identifier, "\n";
>
> # get its corresponding MARC record
> my $response = $ua->get( ROOT . "$identifier/$identifier" . "_meta.mrc" );
> if ( ! $response->is_success ) {
>
> warn $response->status_line;
> next;
>
> }
>
> # save it
> open MARC, " > $identifier.mrc" or die "Can't open $identifier.mrc: $!\n";
> binmode MARC, ":utf8";
> print MARC $response->content;
> close MARC;
>
> }
>
> I then use the venerable marcdump to see the fruits of my labors: marcdump *.mrc. Unfortunately, marcdump returns the following error against (at least) one of my files:
>
> bienfaitsducatho00pina.mrc
> utf8 "\xC3" does not map to Unicode at /System/Library/
> Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162.
>
> What is going on here? Am I saving my files incorrectly? Is the original MARC data inherintly incorrect? Is there some way I can fix the MARC record in question?
>
> --
> Eric Lease Morgan
>
