 |
 |
Jonathan -- thanks so much for tackling this monster. I haven't done anything but run the tests, and while they all pass in MRI 1.9, they all fail under jruby 1.6.7 in 1.9 mode, and one fails (test_read_write(XMLTest)) in 1.8 mode. I'll try to find some time to look and see what you're doing that might be MRI specific. On Thu, Apr 19, 2012 at 5:55 PM, Jonathan Rochkind <[log in to unmask]> wrote: > I have implemented fairly complete and robust proper support for character > encodings in ruby-marc when reading 'binary' marc under ruby 1.9. > > It's currently in a git branch, not yet released, and not yet in git > master. https://github.com/ruby-marc/**ruby-marc/tree/char_encodings<https://github.com/ruby-marc/ruby-marc/tree/char_encodings> > > If anyone who uses this (or doesn't) has a chance to beta test it, it > would be appreciated. One way to test, checkout with git, switch to > 'char_encodings' branch, and `rake install` to install as a gem to your > system. These changes should _only_ effect use under ruby 1.9, and only > effect reading in 'binary' (ISO 2709) marc. > > The new functionality is pretty extensively covered by automated tests, > but there are some weird and complex interactions that can occur depending > on exactly what you're doing, bugs are possible. It was somewhat more > complicated than one might expect to implement a complete solution here, in > part because we _do_ have international users who use ruby-marc, with > encodings that are neither MARC8 nor UTF8, and in fact non-MARC21. > > If any of the other committers (or anyone else) wants to code review, you > are welcome to. > > POSSIBLE BACKWARDS INCOMPAT > > Some previous 0.4.x versions, when running under ruby 1.9 only, would > automatically _transcode_ non-unicode encodings to UTF-8 for you under the > hood. The new version no longer does so automatically (although you can ask > it to). It was not tenable to support that backwards compatibly. > > Everything else _ought_ to be backwards compatible with previous 0.4.x > ruby-marc under ruby 1.9, fixing many problems. > > NEW FEATURES > > All applying to ruby 1.9 only, and to reading binary MARC only. > > * Do a pretty good job of setting encodings properly for your ruby > environment, especially under standard UTF-8 usage. > > * You _can_ and _do have to_ provide an argument for reading non-UTF8 > encodings. (but sadly no support for marc8). > > * You can ask MARC::Reader to transcode to a different encoding when > loading marc. > > * You can ask MARC::Reader to replace bytes that are illegal in the > believed source encoding with a replacement character (or the empty string) > to avoid ruby "invalid UTF-8 byte" exceptions later, and sanitize your > input. > > New features documented in inline comments, see at: > http://rubydoc.info/github/**ruby-marc/ruby-marc/MARC/**Reader<http://rubydoc.info/github/ruby-marc/ruby-marc/MARC/Reader> > > I had trouble making the docs concise, sorry, I think I've been pounding > my head against this stuff so much realizing how complicated it ends up > being that I wasn't sure what to leave out. > -- Bill Dueber Library Systems Programmer University of Michigan Library