 |
 |
> I'm making headway on my MARC records, but only through the use of brute force. > > I used wget to retrieve the MARC records (as well as associated PDF and text files) from the > Internet Archive. I know IA has some bad marc records (and also records w/ bad encoding) from my experience with them in the past. I'm also not sure what the web server / wget will do to the files as well. > I did play a bit with yaz-marcdump to seemingly convert things from marc-8 to utf-8, but I'm not so > sure it does what is expected. Does it actually convert characters, or does it simply change a > value in the leader of each record? If the former, then how do I know it is not double-encoding >things? If the later, then my resulting data set is still broken. There was a bug I seem to remember with yaz-marcdump where it was just toggling the leader. (Or a design flaw where you had to specify a character conversion as well.). But that was fixed a while ago I thought. It's probably one of the better tools out there for this type of stuff. > If MARC records are not well-formed and do not validate according to the standard, then just like > XML processors, they should be used. Garbage in. Garbage out. I'm guessing you meant "they shouldn't be used?" ;). XML processors aren't really known for flexibility in this regard. Unfortunately there's a lot of issues here, not the least of it some of the worse issues I've seen are introduced by well-meaning folks who do things like dump a file out into MARCXML and twiddle with bits or a marc-breaker format and start using tools to dump unicode text into what is really a marc-8 file. Then at some point in the pipeline of conversions enough character encoding conversions happens that the file ends up being messed up. And then there's always the legacy data that got bungled up in the an encoding transfer. I know we've got some bad CJK characters due to this. At some point in converting our marc-8 records one or two characters got mapped to something that's not in the unicode spec at all. At some point we'll clean up those records, you know, when we've got some spare time :P. The problem here has been the tools and they pass whatever internal validations are enforced. Probably more stages need to check for validity, but there's a lot of records that would fail if they did. (I don't even want to think about how many people disable validation, or use the same software stack that generated the marc in the first place, or changes within the marc spec itself over time that makes validation even more difficult. Jon Gorman