 |
 |
For what it's worth, my patch was a stopgap measure, and acknowledged as such at the time. My proposal for a real, comprehensive solution was detailed in a comment in a (now-closed) issue Github ticket[1]. If I'd had the time and the knowledge, I would have implemented it that way. If I'd had the need, I would have made the time and gained the knowledge. As it was, I submitted a patch to make Unicode handling (a) better than it was, and (b) work as well as I needed it to. [1] https://github.com/edsu/pymarc/issues/7#issuecomment-501460 On Thu, Mar 8, 2012 at 12:32 PM, Godmar Back <[log in to unmask]> wrote: > On Thu, Mar 8, 2012 at 3:18 PM, Ed Summers <[log in to unmask]> wrote: > > > Hi Terry, > > > > On Thu, Mar 8, 2012 at 2:36 PM, Reese, Terry > > <[log in to unmask]> wrote: > > > This is one of the reasons you really can't trust the information found > > in position 9. This is one of the reasons why when I wrote MarcEdit, I > > utilize a mixed process when working with data and determining > characterset > > -- a process that reads this byte and takes the information under > > advisement, but in the end treats it more as a suggestion and one part > of a > > larger heuristic analysis of the record data to determine whether the > > information is in UTF8 or not. Fortunately, determining if a set of data > > is in UTF8 or something else, is a fairly easy process. Determining the > > something else is much more difficult, but generally not necessary. > > > > Can you describe in a bit more detail how MARCEdit sniffs the record > > to determine the encoding? This has come up enough times w/ pymarc to > > make it worth implementing. > > > > > One side comment here; while smart handling/automatic detection of > encodings would be a nice feature to have, it would help if pymarc could > operate in an 'agnostic', or 'raw' mode where it would simply preserve the > encoding that's there after a record has been read when writing the record. > > [ Right now, pymarc does not have such a mode - if leader[9] == 'a', the > data is unconditionally utf8 encoded on output as per mbklein's patch. ] > > - Godmar >