Yeah -- this has been part of the MARC standard for quite some time (2004?)...LC added it as a way to protect round trip ability.  MarcEdit has supported this for years -- it's actually one of the questions that I have to answer occasionally when people translate UTF8 code outside of the MARC8 specification back to MARC8.

--tr

-----Original Message-----
From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of Jonathan Rochkind
Sent: Tuesday, November 5, 2013 4:05 PM
To: [log in to unmask]
Subject: [CODE4LIB] a note on MARC8 to UTF8 transcoding: Character references

Do you do sometimes deal with MARC in the MARC8 character encoding?  Do you deal with software that converts from MARC8 to UTF8?

Maybe sometimes you've seen weird escape sequences that look like HTML or XML "character references", like, say "‏".

You, like me, might wonder what the heck that is about -- is it cataloger error, a catalgoer manually entered this or something in error? Is it a software error, some software accidentally stuck this in, at some part in the pipeline?

You can't, after all, just put HTML/XML character references wherever you want -- there's no reason "‏" would mean anything other than &, #, x, 2, etc, when embedded in MARC ISO 2709 binary, right?

Wrong, it turns out!

There is actually a standard that says you _can_ embed XML/HTML-style character references in MARC8, for glyphs that can't otherwise be represented in MARC8. "Lossless conversion [from unicode] to MARC-8 encoding."

http://www.loc.gov/marc/specifications/speccharconversion.html#lossless

Phew, who knew?!

Software that converts from MARC8 to UTF-8 may or may not properly un-escape these character references though. For instance, the Marc4K "AnselToUnicode" class which converts from Marc8 to UTF8 (or other unicode serializations) won't touch these "lossless conversions" (ie, HTML/XML character references), they'll leave them alone in the output, as is.

yaz-marcdump also will NOT un-escape these entities when converting from
Marc8 to UTF8.

So, then, the system you then import your UTF8 records into will now just display the literal HTML/XML-style character reference, it won't know to un-escape them either, since those literals in UTF8 really _do_ just mean & followed by a # followed by an x, etc. It only means something special as a literal in HTML, or in XML -- or it turns out in MARC8, as a 'lossless character conversion'.

So, for instance, in my own Traject software that uses Marc4J to convert from Marc8 to UTF8 -- I'm going to have to go add another pass, that converts HTML/XML-character entities to actual UTF8 serializations.  Phew.

So be warned, you may need to add this to your software too.