 |
 |
On Thu, Mar 8, 2012 at 1:46 PM, Terray, James <[log in to unmask]> wrote: > Hi Godmar, > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 9: > ordinal not in range(128) > > Having seen my fair share of these kinds of encoding errors in Python, I > can speculate (without seeing the pymarc source code, so please don't hold > me to this) that it's the Python code that's not set up to handle the UTF-8 > strings from your data source. In fact, the error indicates it's using the > default 'ascii' codec rather than 'utf-8'. If it said "'utf-8' codec can't > decode...", then I'd suspect a problem with the data. > > If you were to send the full traceback (all the gobbledy-gook that Python > spews when it encounters an error) and the version of pymarc you're using > to the program's author(s), they may be able to help you out further. > > My question is less about the Python error, which I understand, than about the MARC record causing the error and about how others deal with this issue (if it's a common issue, which I do not know.) But, here's the long story from pymarc's perspective. The record has leader[9] == 'a', but really, truly contains ANSEL-encoded data. When reading the record with a MARCReader(to_unicode = False) instance, the record reads ok since no decoding is attempted, but attempts at writing the record fail with the above error since pymarc attempts to utf8 encode the ANSEL-encoded string which contains non-ascii chars such as 0xe8 (the ANSEL Umlaut prefix). It does so because leader[9] == 'a' (see [1]). When reading the record with a MARCReader(to_unicode=True) instance, it'll throw an exception during marc_decode when trying to utf8-decode the ANSEL-encoded string. Rightly so. I don't blame pymarc for this behavior; to me, the record looks wrong. - Godmar (ps: that said, what pymarc does fails in different circumstances - from what I can see, pymarc shouldn't assume that it's ok to utf8-encode the field data if leader[9] is 'a'. For instance, this would double-encode correctly encoded Marc/Unicode records that were read with a MARCReader(to_unicode=False) instance. But that's a separate issue that is not my immediate concern. pymarc should probably remember if a record needs or does not need encoding when writing it, rather than consulting the leader[9] field.) (*) https://github.com/mbklein/pymarc/commit/ff312861096ecaa527d210836dbef904c24baee6