Dan,

I have seen something similar recently.   We (at the University of Virginia) were starting to see records containing the � character.​   I wrote a simple script to take a list of records containing that character, and extract the OCLC number, and then programatically perform a look-up for that OCLC number in WorldCAT via z39.50    The script would then compare our record with the one retrieved from OCLC and replace the fields containing the � character with the corresponding field from the record from OCLC.

For the most part this was working, but in several cases the record retrieved from OCLC contained the � character.    It is not merely an encoding/translation error since multiple different characters have been replaced with that character.   Moreover, in some of the records returned there are some fields that contain correct accent marks while other fields that seemingly ought to contain the same accent mark, the accent mark has the � character substituted in instead.

One example shown below is the record for OCLC number     682540414    (ISBN   9782760521292)

Additionally, I used both my marc4j- based z39.50  libraries and the local copy of yaz-client to retrieve the exact record you used as an example.   The binary form of the record was identical for both methods of retrieving the record.   And there were no instances of the � character in the display of the record.   (shown even further below)


--Robert Haschart
University of Virginia Library


LEADER 03689cam a2200769Mi 4500
001 ocn682540414
003 OCoLC
005 20221102142039.0
008 101122s1996    quc     o     000 0 fre d
006 m     o  d
007 cr |n|---|||||
040   $aEBLCP$beng$epn$cEBLCP$dOCLCQ$dAUW$dOCLCO$dP@U$dOCLCF$dOCLCO$dOCLCQ$dCNKUC$dFVL$dOCLCQ$dOCLCA$dOCLCQ$dOCLCO$dOCLCQ$dJSTOR$dOCLCQ$dOCLCO$dQGK$dOCLCO$dOCL
066   $cZsym
016   $z96940719X
019   $a1259108319
020   $a9782760521292
020   $a276052129X
020   $a1441600825
020   $a9781441600820
035   $a(OCoLC)682540414$z(OCoLC)1259108319
037   $a22573/ctv18n0fbx$bJSTOR
050  4$aHD938
072  7$aBUS$x043000$2bisacsh
082 04$a338.4/791
100 1 $aLaplante, Marc.
245 12$aL'expérience touristique contemporaine :$bFondements sociaux et culturels.
260   $aQuébec :$bPresses de l''niversite du Quebec,$c1996.
300   $a1 online resource (204 pages)
336   $atext$btxt$2rdacontent
337   $acomputer$bc$2rdamedia
338   $aonline resource$bcr$2rdacarrier
490 1 $aCollection Tourisme
505 0 $aTable des matières; Liste des tableaux; Liste des figures; Introduction; Partie 1_L'enracinement du tourisme dans la socioté moderne; Chapitre 1_Touriste et tourisme d'hier et d'aujourd'hui; Chapitre 2_Sociétés et cultures de l'Occident industrialisé; Partie 2_Les dimensions de la culture touristique actuelle; Chapitre 3_L'expérience touristique comme expérience culturelle; Chapitre 4_La culture touristique personnelle; Chapitre 5_La culture touristique institutionnalisée; Conclusion; Guide bibliographique.
520   $aPour comprendre le tourisme comme fait social et culturel typique des sociétés postindustrielles, ce livre présente d'abord les principaux rapports entre tourisme et société puis élabore le concept de culture touristique.
588 0 $aPrint version record.
504   $aComprend des réf. bibliogr.
546   $aFrench.
650  0$aTourism and the arts.
650  0$aTourism$xSocial aspects.
650  0$aTourism.
650  0$aCulture.
650  0$aHeritage tourism.
650  6$aCulture$xAspect sociologique.
650  6$aTourisme culturel.
650  6$aTourisme$xAspect social.
650  6$aTourisme.
650  6$aTourisme et arts.
650  7$acultural tourism.$2aat
650  7$atourism.$2aat
650  7$aBUSINESS & ECONOMICS$xMarketing$xGeneral.$2bisacsh
650  7$aHeritage tourism.$2fast$0(OCoLC)fst00955465
650  7$aCulture.$2fast$0(OCoLC)fst00885059
650  7$aTourism.$2fast$0(OCoLC)fst01153142
650  7$aTourism and the arts.$2fast$0(OCoLC)fst01153221
650  7$aTourism$xSocial aspects.$2fast$0(OCoLC)fst01153202
655  4$aElectronic books.
776 08$iPrint version:$aLaplante, Marc.$tExpérience touristique contemporaine : Fondements sociaux et culturels.$dQuébec : Presses de l''niversite du Quebec, ©1996$z9782760509108
830  0$aTourisme (Presses de l'Université du Québec)
856 40$3desLibris$uhttps://www.deslibris.ca/ID/422431
856 40$3Project MUSE$uhttps://muse.jhu.edu/books/9782760521292
856 40$3ProQuest Ebook Central$uhttps://public.ebookcentral.proquest.com/choice/publicfullrecord.aspx?p=600242
856 40$3JSTOR$uhttps://www.jstor.org/stable/10.2307/j.ctv18pgzcc
880  0$6830-00/Zsym$aCollection Tourisme (Presses de l'Universit�e du Qu�ebec)
938   $aProQuest Ebook Central$bEBLB$nEBL600242
938   $aProject MUSE$bMUSE$nmuse20945
994   $aZ0$bVA@
948   $hHELD BY VA@ - 86 OTHER HOLDINGS

LEADER 02823cam a2200721 a 4500
001 ocn276644602
003 OCoLC
005 20221102144112.0
008 081121s2008    quca     b    000 p eng
040   $aNLC$beng$cNLC$dCDX$dBWX$dSUC$dUKMGB$dOCLCF$dNLC$dOCLCA$dREB$dOCLCA$dOCLCQ$dOCLCO
015   $a20089424808E$2can
015   $a20089424808$2can
016   $a(AMICUS)000034543125
016 7 $a015299625$2Uk
020   $a9782920325241$q(Galerie de l'UQAM)
020   $a2920325248
020   $a9782980985249$q(Galerie Simon Blais)
020   $a2980985244
035   $a(OCoLC)276644602
041 0 $aeng$afre
043   $an-cn-qu
050  4$aN6549.G686$bA4 2008
055 02$aNB249*
055  3$aN6549 G686$bA4 2009
082 04$a730.92$222
084   $acci1icc$2lacc
100 1 $aDéry, Louise,$d1955-
245 10$aRêver le nouveau monde :$bune œuvre d'art public de Michel Goulet offerte par la ville de Montréal à la ville de Québec dans le cadre des célébrations du 400e anniversaire de sa fondation /$ctextes, Louise Déry et Michel Goulet.
246 1 $iAdded title page in English:$aDreaming a new world
260   $a[Montréal] :$bGalerie Simon Blais,$c2008.
300   $a69 pages :$billustrations (some color) ;$c20 cm
336   $atext$btxt$2rdacontent
337   $aunmediated$bn$2rdamedia
338   $avolume$bnc$2rdacarrier
500   $aIncludes poems.
500   $aTranslation: Donald Pistolesi.
504   $aIncludes bibliographical references (pages 68-69).
546   $aText in French and English.
550   $aCo-published by: City of Montréal and Galerie de l'UQAM.
600 10$aGoulet, Michel,$d1944-
650  0$aPublic art$zQuébec (Province)$zQuébec.
650  0$aSculpture, French-Canadian$zQuébec (Province)$y21st century.
650  5$aSculpture, French Canadian$zQuébec (Province)$y21st century.
650  6$aSculpture québécoise$y21e siècle.
600 17$aGoulet, Michel,$d1944-$2fast$0(OCoLC)fst00070599
650  7$aPublic art.$2fast$0(OCoLC)fst01082064
650  7$aSculpture, French-Canadian.$2fast$0(OCoLC)fst01895320
651  7$aQuébec.$2fast$0(OCoLC)fst01207316
651  7$aQuébec$zQuébec.$2fast$0(OCoLC)fst01206798
648  7$a2000-2099$2fast
700 1 $aGoulet, Michel,$d1944-
710 2 $aGalerie de l'UQAM.
710 2 $aGalerie Simon Blais.
710 1 $aMontréal (Québec)
856 42$3French equivalent / Équivalent français$uhttps://bac-lac.on.worldcat.org/oclc/276644605
938   $aCoutts Information Services$bCOUT$n9265731
029 0 $aNLC$b000034543125
029 0 $aNLC$b20089424808E
029 1 $aBWX$bR3781137
029 1 $aCDX$b9265731
029 1 $aUKMGB$b015299625
994   $aZ0$bVA@
948   $hNO HOLDINGS IN VA@ - 21 OTHER HOLDINGS

________________________________
From: Code for Libraries <[log in to unmask]> on behalf of Dan Scott <[log in to unmask]>
Sent: Tuesday, November 1, 2022 11:32 AM
To: [log in to unmask] <[log in to unmask]>
Subject: [CODE4LIB] Connecting to OCLC WorldCat Z39.50 servers: character encoding issues

Hello:

I'm wondering if anyone else has run into this situation with OCLC WorldCat
Z39.50 servers. When we search Voilà (Canada's national union catalogue,
built on WorldCat)
<https://library-archives.canada.ca/eng/collection/basics/pages/voila.aspx>
via Z39.50 for a record that is catalogued in French for ILL purposes, the
resulting records contain corrupted characters where accented characters
are expected. We're seeing this in Ex Libris Alma, where Voilà is an
external search source maintained by Alma (we can't see the parameters Alma
is using to connect).

For example, searching Voilà for ISBN 9782920325241 returns a list of
records for which the summaries display without accents. However, checking
the MARC record itself shows that where there should be accents, there are
instead literals in place of Unicode codepoints.

The 245 for the first record returned for ISBN 9782920325241 appears as
follows in Alma:
245 1 0 $$a R<U+00fd>ever le nouveau monde : $$b une <U+00fd>uvre d'art
public de Michel Goulet offerte par la ville de Montr<U+00fd>eal <U+00fd>a
la ville de Qu<U+00fd>ebec dans le cadre des c<U+00fd>el<U+00fd>ebrations
du 400e anniversaire de sa fondation / $$c textes, Louise D<U+00fd>ery et
Michel Goulet.

   - Where we see R<U+00fd>ever, we should see Rèver.
   - Where we see <U+00fd>uvre, we should see œuvre.
   - Where we see Montr<U+00fd>eal, we should see Montréal.
   - Where we see <U+00fd>a, we should see à.

I don't *think *this is an example of a Unicode canonicalization into a
normalized form using combining characters, as <U+00fd> is not a combining
character. If that was the case, we would see <U+0301>e for é, <U+0300>a
for à, etc.

Instead, it looks like a case of corruption introduced by incorrect
decoding. This can occur when the Z39.50 source uses UTF8 and the client
interprets it as MARC8, or vice versa.

That said, I have tested a few different approaches to querying Voilà using
the yaz-client command line Z39.50 client (version 5.28.0 on Ubuntu), and
it seems that Voilà itself might be returning the corrupted characters.

For example, when I set the negotiation character set to MARC-8 (matching
the LDR[09] of the returned record) and the display character set to UTF-8,
the results on the screen and in the file show corruption:

$ yaz-client -m testvoila.mrc
Z> negcharset MARC-8
Character set negotiation : MARC-8
Z> charset
Negotiation character set `MARC-8'
Records in charset yes
Charneg version 3
Display character set is `UTF-8'
MARC character set is `none'
Query character set is `none'
Z> authentication USERNAME/PASSWORD
Authentication set to Open (USERNAME/PASSWORD)
Z> open fsz3950.oclc.org:210/gclacConnecting...OK.
Sent initrequest.
Connection accepted by v3 target.
UserInformationfield:
{
  OID: 1 2 840 10003 10 1000 17 1
  {
    ANY (len=66)
  }
}
OCLC UserInformation:
{
  motd 'You are searching the FirstSearch 5.0 Z39.50 Server!'
  dblist {
    'laccat'
  }
}
Options: search present delSet triggerResourceCtrl scan sort
extendedServices namedResultSets
Elapsed: 0.167063
Z> find @attr 1=7 9782920325241
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 6, setno 1
records returned: 0
Elapsed: 0.190875
Z> show
Sent presentRequest (1+1). Records: 1 [laccat]
Record type: USmarc
02802cam 2200685 a 4500 008 20081121s2008 quca b 000 p eng
015 $a 20089424808E $2 can
015 $a 20089424808 $2 can
016 $a (AMICUS)000034543125
016 7 $a 015299625 $2 Uk
020 $a 9782920325241 $q (Galerie de l'UQAM)
020 $a 2920325248
020 $a 9782980985249 $q (Galerie Simon Blais)
020 $a 2980985244
040 $a NLC $b eng $c NLC $d CDX $d BWX $d SUC $d UKMGB $d OCLCF $d NLC $d
OCLCA $d REB $d OCLCA $d OCLCQ $d OCLCO
041 0 $a eng $a fre
043 $a n-cn-qu 050 4 $a N6549.G686 $b A4 2008
055 02 $a NB249*
055 3 $a N6549 G686 $b A4 2009
082 04 $a 730.92 $2 22
084 $a cci1icc $2 lacc
100 1 $a D�ery, Louise, $d 1955- 245 10 $a R�ever le nouveau monde : $b une
�uvre d'art public de Michel Goulet offerte par la ville de Montr�eal �a la
ville de Qu�ebec dans le cadre des c�el�ebrations du 400e anniversaire de
sa fondation / $c textes, Louise D�ery et Michel Goulet.

In the file with the downloaded record (looking at it with Vim in binary
mode) you can see the same <fd> characters as we see in Alma for the same
record:

02802cam 2200685 a
4500008004300000015002200043015002100065016002500086016001800111020003900129020001500168020004100183020001500224040008500239041001300324043001200337050002400349055001100373055002400384082001500408084001800423100002600441245024000467246005500707260004700762300005200809336002600861337002800887338002700915500002000942500003500962504005500997546003201052550006201084600002701146600003301173648002001206650004501226650006601271650006601337650004101403650002201444650003801466651001301504651002201517700002701539710002301566710002501589710002401614856009401638938004701732945000801779947003001787948014101817948003801958948004701996948002902043948003202072949001202104^^20081121s2008
quca b 000 p eng ^^ ^_a20089424808E^_2can^^ ^_a20089424808^_2can^^
^_a(AMICUS)000034543125^^7 ^_a015299625^_2Uk^^ ^_a9782920325241^_q(Galerie
de l'UQAM)^^ ^_a2920325248^^ ^_a9782980985249^_q(Galerie Simon Blais)^^
^_a2980985244^^
^_aNLC^_beng^_cNLC^_dCDX^_dBWX^_dSUC^_dUKMGB^_dOCLCF^_dNLC^_dOCLCA^_dREB^_dOCLCA^_dOCLCQ^_dOCLCO^^0
^_aeng^_afre^^ ^_an-cn-qu^^ 4^_aN6549.G686^_bA4 2008^^02^_aNB249*^^
3^_aN6549 G686^_bA4 2009^^04^_a730.92^_222^^ ^_acci1icc^_2lacc^^1
^_aD<fd>ery, Louise,^_d1955-^^10^_aR<fd>ever le nouveau monde :^_bune
<fd>uvre d'art public de Michel Goulet offerte par la ville de Montr<fd>eal
<fd>a la ville de Qu<fd>ebec dans le cadre des c<fd>el<fd>ebrations du 400e
anniversaire de sa fondation /^_ctextes, Louise D<fd>ery et Michel Goulet.

If anyone has run into this with OCLC WorldCat Z39.50 and figured out a
solution, I would love to hear it. It would even be helpful to know if you
have a connection to an OCLC WorldCat Z39.50 server and aren't seeing these
kinds of problems. I know most of you probably won't have authentication
credentials for Voilà, but the setup has to be the same for other OCLC
WorldCat Z39.50 servers... right?

In Z39.50 and character encoding madness, I remain,
Dan