Hi list,
I am a Metadata librarian but not a programmer, sorry if my question seems naïve. We use XSLT stylesheet to transform some harvested DC records from DSpace to MARC in MarcEdit, and then export them to OCLC.
Some characters do not display correctly and need manual editing, for example:
In MarcEditor                                                               	Transferred to OCLC	                  Edit in OCLC
Bayes’ theorem                                                          	Bayes⁰́₉ theorem   	                  Bayes' theorem
―it won‘t happen here‖ attitude        	⁰́₅it won⁰́₈t happen here⁰́₆ attitude    	"it won't happen here" attitude
“Generation Y”                                            	⁰́₋Generation Y⁰́₊                                      	"Generation Y"
listeners‟ evaluations             	              listeners⁰́Ÿ evaluations                                  	listeners' evaluations
high school – from	                               high school ⁰́₃ from   	                                                  high school – from
Co₀․₅Zn₀․₅Fe₂O₄	                             Co²́⁰⁰́Þ²́⁵Zn²́⁰⁰́Þ²́⁵Fe²́²O²́⁴	                                   Co0.5Zn0.5Fe2O4?
μ	                                                               Îơ	                                                                                           μ
Nafion®	                                               Nafion℗ʼ	                                                                         Nafion®
Lévy	                                                                  L©♭vy	                                                                           Lévy
43±13.20 years	                                               43℗ł13.20 years	                                                43±13.20 years   
12.6 ± 7.05 ft∙lbs	                            12.6 ℗ł 7.05 ft⁸́₉lbs	                                               12.6 ± 7.05 ft•lbs
‘Pouring on the Pounds'  	         ⁰́₈Pouring on the Pounds'	                        'Pouring on the Pounds'  
k-ε turbulence	                                             k-Îæ turbulence	                                             k-ε turbulence
student—neither parents	        student⁰́₄neither parents	                        student-neither parents
Λ = M – {p1, p2,…,pκ}	                        Î₎ = M ⁰́₃ {p1, p2,⁰́Œ,pÎð}	                              ? (won’t save)
M = (0, δ)x × Y	                                           M = (0, Îþ)x ©₇ Y	                                                ?     
100°	                                                        100℗ð	                                                               100⁰
(α ≥16º)	                                       (Îł ⁹́Æ16℗ð)                                          	                (α>=16⁰)
naïve	                                                        na©¯ve	                                                                  naïve

To deal with this, we normally replace limited numbers of characters in MarcEditor first and then do the compiling and transfer. For example: replace ’ to ', “ to ", ” to " and ‟ to '. I am not sure about the right and efficient way to solve this problem. I see that the XSLT stylesheet specifies encoding="UTF-8". Is there a systematic way to make the character transform and display right? Thank you for your suggestion and feedback!


-----Original Message-----
From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of Tod Olson
Sent: Tuesday, April 17, 2012 10:13 PM
To: [log in to unmask]
Subject: Re: [CODE4LIB] more on MARC char encoding: Now we're about ISO_2709 and MARC21

In practice it seems to mean UTF-8. At least I've only seen UTF-8, and I can't imagine the code that processes this stuff being safe for UTF-16 or UTF-32. All of the offsets are byte-oriented, and there's too much legacy code that makes assumption about null-terminated strings.


On Apr 17, 2012, at 6:55 PM, Jonathan Rochkind wrote:

> Okay, forget XML for a moment, let's just look at marc 'binary'.
> First, for Anglophone-centric MARC21.
> The LC docs don't actually say quite what I thought about leader byte 09, used to advertise encoding:
> a - UCS/Unicode
> Character coding in the record makes use of characters from the Universal Coded Character Set (UCS) (ISO 10646), or Unicode™, an industry subset.
> That doesn't say UTF-8. It says UCS or "Unicode". What does that actually mean?  Does it mean UTF-8, or does it mean UTF-16 (closer to what used to be called "UCS" I think?).  Whatever it actually means, do people violate it in the wild?
> Now we get to non-Anglophone centric marc. I think all of which is ISO_2709?  A standard which of course is not open access, so I can't get it to see what it says.
> But leader 09 being used for encoding -- is that Marc21 specific, or is it true of any ISO-2709?  Marc8 and "unicode" being the only valid encodings can't be true of any ISO-2709, right?
> Is there a generic ISO-2709 way to deal with this, or not so much?