On 11/20/13 11:40 AM, Scott Prater wrote:
> Not sure what the details of our issue was on Monday -- but we do have
> records that are supposedly encoded in UTF-8, but nonetheless contain
> invalid characters.

Oh, and I'd clarify, if you haven't figured it out already, if those are 
ISO 2709 binary records, you can ask the reader to do different things 
there in that case (already avail in current ruby-marc release):

# raise:
MARC::Reader("something.marc", :validate_encoding => true)

# replace with unicode replacement char:
MARC::Reader("something.marc", :invalid => :replace)

This is already available in present ruby-marc release.

I would suggest one or the other -- the default of leaving bad bytes in 
your ruby strings is asking for trouble, and you probably don't want to 
do it, but was made the default for backwards compat reasons with older 
versions of ruby-marc. (See why I am reluctant to add another default 
that we don't think hardly anyone would actually want? :) )

Oh, and you may also want to explicitly specify the expected encoding to 
avoid confusing:

MARC::Reader("something.marc", :external_encoding => "UTF-8", 
:validate_encoding => true)

(It will also work with any other encoding recognized by ruby, for those 
with legacy, possibly international, data).

This stuff is confusing to explain, there are so many permutations and 
combinations of circumstances involved.  But I'll try to improve the 
ruby-marc docs on this stuff, as part of adding the yet more options for 
MARC8 handling.