Do you do sometimes deal with MARC in the MARC8 character encoding? Do
you deal with software that converts from MARC8 to UTF8?
Maybe sometimes you've seen weird escape sequences that look like HTML
or XML "character references", like, say "‏".
You, like me, might wonder what the heck that is about -- is it
cataloger error, a catalgoer manually entered this or something in
error? Is it a software error, some software accidentally stuck this in,
at some part in the pipeline?
You can't, after all, just put HTML/XML character references wherever
you want -- there's no reason "‏" would mean anything other than
&, #, x, 2, etc, when embedded in MARC ISO 2709 binary, right?
Wrong, it turns out!
There is actually a standard that says you _can_ embed XML/HTML-style
character references in MARC8, for glyphs that can't otherwise be
represented in MARC8. "Lossless conversion [from unicode] to MARC-8
Phew, who knew?!
Software that converts from MARC8 to UTF-8 may or may not properly
un-escape these character references though. For instance, the Marc4K
"AnselToUnicode" class which converts from Marc8 to UTF8 (or other
unicode serializations) won't touch these "lossless conversions" (ie,
HTML/XML character references), they'll leave them alone in the output,
yaz-marcdump also will NOT un-escape these entities when converting from
Marc8 to UTF8.
So, then, the system you then import your UTF8 records into will now
just display the literal HTML/XML-style character reference, it won't
know to un-escape them either, since those literals in UTF8 really _do_
just mean & followed by a # followed by an x, etc. It only means
something special as a literal in HTML, or in XML -- or it turns out in
MARC8, as a 'lossless character conversion'.
So, for instance, in my own Traject software that uses Marc4J to convert
from Marc8 to UTF8 -- I'm going to have to go add another pass, that
converts HTML/XML-character entities to actual UTF8 serializations. Phew.
So be warned, you may need to add this to your software too.