 |
 |
I have implemented fairly complete and robust proper support for character encodings in ruby-marc when reading 'binary' marc under ruby 1.9. It's currently in a git branch, not yet released, and not yet in git master. https://github.com/ruby-marc/ruby-marc/tree/char_encodings If anyone who uses this (or doesn't) has a chance to beta test it, it would be appreciated. One way to test, checkout with git, switch to 'char_encodings' branch, and `rake install` to install as a gem to your system. These changes should _only_ effect use under ruby 1.9, and only effect reading in 'binary' (ISO 2709) marc. The new functionality is pretty extensively covered by automated tests, but there are some weird and complex interactions that can occur depending on exactly what you're doing, bugs are possible. It was somewhat more complicated than one might expect to implement a complete solution here, in part because we _do_ have international users who use ruby-marc, with encodings that are neither MARC8 nor UTF8, and in fact non-MARC21. If any of the other committers (or anyone else) wants to code review, you are welcome to. POSSIBLE BACKWARDS INCOMPAT Some previous 0.4.x versions, when running under ruby 1.9 only, would automatically _transcode_ non-unicode encodings to UTF-8 for you under the hood. The new version no longer does so automatically (although you can ask it to). It was not tenable to support that backwards compatibly. Everything else _ought_ to be backwards compatible with previous 0.4.x ruby-marc under ruby 1.9, fixing many problems. NEW FEATURES All applying to ruby 1.9 only, and to reading binary MARC only. * Do a pretty good job of setting encodings properly for your ruby environment, especially under standard UTF-8 usage. * You _can_ and _do have to_ provide an argument for reading non-UTF8 encodings. (but sadly no support for marc8). * You can ask MARC::Reader to transcode to a different encoding when loading marc. * You can ask MARC::Reader to replace bytes that are illegal in the believed source encoding with a replacement character (or the empty string) to avoid ruby "invalid UTF-8 byte" exceptions later, and sanitize your input. New features documented in inline comments, see at: http://rubydoc.info/github/ruby-marc/ruby-marc/MARC/Reader I had trouble making the docs concise, sorry, I think I've been pounding my head against this stuff so much realizing how complicated it ends up being that I wasn't sure what to leave out.