On 11/20/2013 11:18 AM, Jonathan Rochkind wrote:
> On 11/20/13 11:40 AM, Scott Prater wrote:

> I would suggest one or the other -- the default of leaving bad bytes in
> your ruby strings is asking for trouble, and you probably don't want to
> do it, but was made the default for backwards compat reasons with older
> versions of ruby-marc. (See why I am reluctant to add another default
> that we don't think hardly anyone would actually want? :) )

Thanks for your usage suggestions and work on this, Jonathan.  I work 
mostly with marc4j, not ruby-marc, so I'm pretty unfamiliar with the 
capabilities of the gem.  My comments are more oriented towards general 
error handling when processing MARC streams.

I think the issue comes down to a distinction between a stream and a 
record.  Ideally, the ruby-marc library would keep pointers to which 
record it is in, where the record begins, and where the record ends in 
the stream.  If a valid header and end-of-record delimiter are in place, 
then the library should be able to reject the record if it contains 
garbage in between those two points, without compromising the integrity 
of the entire stream.  So my final output would not contain bad data; 
it would simply be missing some records, records that contained bad data.

Here's some (partial) pseudo ruby code of how I'd like to handle it:

count=0
reader = MARC::Reader.new('marc8.dat')
writer = MARC::XMLWriter.new('marc-utf8.xml')
for record in reader
   count+=1
   begin
      utf8rec = record.convert_to_utf()
      writer.write(utf8rec)
   rescue => exception
      log exception, "Skipping record #{count}"
   end
   ... now read the next record...
end

This example doesn't capture the exception if the next record can't be 
retrieved, because the stream is corrupt, but that would be the other 
addition I'd make.  The larger point is that reading a MARC stream 
should be handled as reading a sequence of MARC records encoded in that 
stream -- one bad record does not automatically invalidate the entire 
stream; it only invalidates it if the next record can't be found.

-- Scott

-- 
Scott Prater
Shared Development Group
General Library System
University of Wisconsin - Madison
[log in to unmask]
5-5415