I suspect it's been brought up, but that the only solutions implemented were for individual institutions.
At the Center for Jewish History, we had a similar character encoding problem some years ago when we were doing a holdings reclamation project (back when you got one for free). I don't recall the details, as it was about five years ago & at a different job, but OCLC spent months writing custom code to handle our records, which had A LOT of special characters given the language of the materials. The characters were breaking somewhere in the round trip -- we'd upload what looks like good records to WorldShare from Aleph, but when we got them back the characters were broken. They did solve it eventually, but it looks like it was just for our problem, not a general solution.
Jaime Taylor
Discovery & Resource Management Systems Coordinator
W.E.B. Du Bois Library
University of Massachusetts
-----Original Message-----
From: Code for Libraries <[log in to unmask]> On Behalf Of Eric Phetteplace
Sent: Thursday, March 9, 2023 1:44 PM
To: [log in to unmask]
Subject: Re: [CODE4LIB] Connecting to OCLC WorldCat Z39.50 servers: character encoding issues
Hi Dan,
I can confirm that we (a U.S. institution) are experiencing the same problems. We have OCLC's z39.50 server in our Koha catalog but imported records with accented characters are garbled. OCLC says <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.oclc.org%2FMetadata_Services%2FZ3950_Cataloging%2FGet_started%2FConfiguration_guide_for_OCLC_Z39.50_Cataloging%23Z39.50_services_supported&data=05%7C01%7Cjaimetaylor%40umass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33954dc194bbd0b2e56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sQ84qQLoRq2Yum9r4QuDN%2BrOXrUZLy6l4BfNXvmKCO4%3D&reserved=0>
they support UTF-8 too but changing the character encoding didn't fix things for us, we just got *different* garbled characters.
I assume this must have worked in the past, no? Someone would have brought it up if such a major source of metadata couldn't handle accented characters.
Best,
Eric Phetteplace
Systems Librarian
California College of the Arts
On Thu, Mar 9, 2023 at 9:39 AM Dan Scott <[log in to unmask]> wrote:
> Following up on this thread, because it's still a problem for us.
>
> If it were just a few records, and the records displayed the same
> corruption in the discovery interface as in the Z39.50 interface, then
> Occam's razor would indeed suggest the blame lies with those who
> uploaded the records.
>
> But that's not the case. In the human-facing Voilà (WorldCat
> Discovery) interface, accented characters in records are (for the most
> part) displayed correctly.
>
> On the other hand, I can't find a single record via Voilà's Z39.50
> server that displays an accented character correctly. It seems
> unlikely that every Canadian library has only uploaded corrupted
> records to WorldCat, and that OCLC has someone figured out how to
> display those corrupted records correctly to humans. Rather, Occam's
> razor strongly suggests that OCLC's Voilà Z39.50 server implementation is to blame.
>
> For example, OCLC # 276644602
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcan
> ada.on.worldcat.org%2Foclc%2F276644602&data=05%7C01%7Cjaimetaylor%40um
> ass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33954dc194bbd0b2e
> 56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
> C4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
> %7C&sdata=vISf%2BGPsBgqH0QItin3ysEbDCh7bD6D3pq35D5vE2Q8%3D&reserved=0>
> in Voilà (WorldCat
> Discovery) displays perfectly, whereas the same record retrieved via
> Voilà's Z39.50 server is garbage.
>
> The problem isn't just with non-ASCII characters. The 008 is wrong in
> EVERY record. Voilà's Z39.50 server generates a 008 beginning with
> YYYYMMDD when the date is limited to character positions 00-05 (should just be YYYYMM).
> That breaks everything else in the 008. Every record has this problem.
> I've included an example below.
>
> So per my previous emails, I suspect that OCLC's process to strip the
> precious fields 001, 003, and 035 corrupts the MARC records available
> through Voilà's Z39.50 server. Given that the WorldCat Discovery
> interface displays the records without corruption, it seems likely
> that it's a run-time filter that introduces both the laughable date in
> 008 and the corrupted non-ASCII characters.
>
> It's mind-boggling, because OCLC knows how to implement Z39.50, MARC8 vs.
> UTF-8 encodings, and how to validate MARC records. (Thanks to
> IndexData, many of us could stand up a Z39.50 server for a few million
> MARC records over a lunch break. Give us a whole day, and we could
> implement the run-time field-stripping and a test suite to validate
> that the records weren't corrupted!)
>
> If one were conspiratorially minded, one might hypothesize that OCLC
> deliberately broke Z39.50 for Voilà. Why? Every Canadian library can
> get free access, which opens up the possibility of copy cataloguing a
> significant subset of OCLC records without having to pay for a
> WorldCat cataloguing subscription.
>
> I would still be interested in the results from other countries that
> have implemented a national union catalogue via OCLC, just to see if
> this is a Canada-only thing, or a more widespread problem.
>
> Using yaz-client to pull up the same record that Robert found had no
> corrupted characters (skipping the connection details):
>
> Z> find @attr 1=12 276644602
> Sent searchRequest.
> Received SearchResponse.
> Search was a success.
> Number of hits: 1, setno 3
> records returned: 0
> Elapsed: 1.756144
> Z> show
> Sent presentRequest (1+1).
> Records: 1
> [laccat]Record type: USmarc
> 02802cam 2200685 a 4500
> 008 20081121s2008 quca b 000 p eng
> 015 $a 20089424808E $2 can
> 015 $a 20089424808 $2 can
> 016 $a (AMICUS)000034543125
> 016 7 $a 015299625 $2 Uk
> 020 $a 9782920325241 $q (Galerie de l'UQAM)
> 020 $a 2920325248
> 020 $a 9782980985249 $q (Galerie Simon Blais)
> 020 $a 2980985244
> 040 $a NLC $b eng $c NLC $d CDX $d BWX $d SUC $d UKMGB $d OCLCF $d NLC
> $d OCLCA $d REB $d OCLCA $d OCLCQ $d OCLCO
> 041 0 $a eng $a fre
> 043 $a n-cn-qu
> 050 4 $a N6549.G686 $b A4 2008
> 055 02 $a NB249*
> 055 3 $a N6549 G686 $b A4 2009
> 082 04 $a 730.92 $2 22
> 084 $a cci1icc $2 lacc
> 100 1 $a D ery, Louise, $d 1955-
> 245 10 $a R ever le nouveau monde : $b une uvre d'art public de
> Michel Goulet offerte par la ville de Montr eal a la ville de Qu ebec
> dans le cadre des c el ebrations du 400e anniversaire de sa fondation
> / $c textes, Louise D ery et Michel Goulet.
> 246 1 $i Added title page in English: $a Dreaming a new world
> 260 $a [Montr eal] : $b Galerie Simon Blais, $c 2008.
> 300 $a 69 pages : $b illustrations (some color) ; $c 20 cm
> 336 $a text $b txt $2 rdacontent
> 337 $a unmediated $b n $2 rdamedia
> 338 $a volume $b nc $2 rdacarrier
> 500 $a Includes poems.
> 500 $a Translation: Donald Pistolesi.
> 504 $a Includes bibliographical references (pages 68-69).
> 546 $a Text in French and English.
> 550 $a Co-published by: City of Montr eal and Galerie de l'UQAM.
> 600 10 $a Goulet, Michel, $d 1944-
> 600 17 $a Goulet, Michel, $d 1944- $2 fast
> 648 7 $a 2000-2099 $2 fast
> 650 0 $a Public art $z Qu ebec (Province) $z Qu ebec.
> 650 0 $a Sculpture, French-Canadian $z Qu ebec (Province) $y 21st century.
> 650 5 $a Sculpture, French Canadian $z Qu ebec (Province) $y 21st century.
> 650 6 $a Sculpture qu eb ecoise $y 21e si ecle.
> 650 7 $a Public art. $2 fast
> 650 7 $a Sculpture, French-Canadian. $2 fast
> 651 7 $a Qu ebec.
> 651 7 $a Qu ebec $z Qu ebec.
> 700 1 $a Goulet, Michel, $d 1944-
> 710 2 $a Galerie de l'UQAM.
> 710 2 $a Galerie Simon Blais.
> 710 1 $a Montr eal (Qu ebec)
> 856 42 $3 French equivalent / Equivalent fran cais $u
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbac-lac.on.worldcat.org%2Foclc%2F276644605&data=05%7C01%7Cjaimetaylor%40umass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33954dc194bbd0b2e56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Dc3qmAj63zqNVlYfk69DUf8NVjPelsqmxxNrIFqf3rE%3D&reserved=0
> 938 $a Coutts Information Services $b COUT $n 9265731
> 945 $f bks
> 947 $n 20081121 $p 20220519 $m 21 $s N
> 948 $a CA $b ON $c caago $d ART GALLERY OF ONTARIO $c NLC $d LIBRARY AND
> ARCHIVES CANADA $c toh $d TORONTO PUB LIBR $c u3g $d UNIV OF GUELPH $c
> YOU $d YORK UNIV LIBR
> 948 $a CA $b AB $c UAU $d UNIV OF CALGARY LIBR
> 948 $a CA $b BC $c UBC $d UNIV OF BRITISH COLUMBIA LIBR
> 948 $a CA $b QC $c LGG $d MCGILL UNIV
> 948 $a CA $b SK $c REB $d UNIV OF REGINA
> 949 $i b $x b $h ?
>
>
> On Wed, Nov 2, 2022 at 6:45 PM <[log in to unmask]> wrote:
>
> > Most likely, whoever is uploading or editing these records is
> > including (or in the case of editing, mangling) incorrect character
> > sequences. I downloaded the data in MARC8 so I can see the bytes,
> > and the block characters you are seeing are &xFFFD; -- which OCLC
> > would use as a replacement character to stand in for an incorrect or
> > unknown byte sequence. My guess, the records are uploaded in UTF8
> > and whatever is creating or editing these records is mangling some of the characters.
> >
> > --tr
> >
> > -----Original Message-----
> > From: Code for Libraries <[log in to unmask]> On Behalf Of Dan
> Scott
> > Sent: Wednesday, November 2, 2022 4:08 PM
> > To: [log in to unmask]
> > Subject: Re: [CODE4LIB] Connecting to OCLC WorldCat Z39.50 servers:
> > character encoding issues
> >
> > Robert, thanks so much for replying with your findings! Super helpful.
> >
> > Based on the record you shared, I noted a few differences between
> > what
> I'm
> > seeing with Voilà and "real" WorldCat:
> >
> > - Voilà does not include OCLC numbers; the 001, 003, and 035 are
> > stripped (likely other fields as well)
> > - Voilà returns records encoded in MARC8 whereas WorldCat returns
> > records encoded in UTF8
> >
> > I tried more searches against Voilà's Z39.50 server. In every case
> > where there should be an accented character, the same corruption occurs.
> >
> > I also noticed that WorldCat's Z39.50 server uses the hostname
> > zcat.oclc.org and is documented quite differently <
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp
> .oclc.org%2FMetadata_Services%2FZ3950_Cataloging%2FGet_started%2FConfi
> guration_guide_for_OCLC_Z39.50_Cataloging&data=05%7C01%7Cjaimetaylor%4
> 0umass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33954dc194bbd0
> b2e56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=ABrlK%2FZmDYqQ5iBP%2BM4H3EnaVAWWDN77idU5yF8WNGE%3D&reserv
> ed=0
> > >
> > from FirstSearch's Z39.50 service, which uses the hostname
> > fsz3950.oclc.org.
> > The documentation for FirstSearch Z39.50 <
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp
> .oclc.org%2FDiscovery_and_Reference%2FFirstSearch%2FZ3950_access%2FZ39
> 50_configuration_for_FirstSearch_databases&data=05%7C01%7Cjaimetaylor%
> 40umass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33954dc194bbd
> 0b2e56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTWFpbGZsb3d8eyJWI
> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7
> C%7C%7C&sdata=tZPDt5Gr1w%2B%2B%2B%2BcajNlm0TWmfUyyqgSmEefVCbTWWfc%3D&r
> eserved=0
> > >
> > doesn't even mention encodings, which seems like quite a red flag in
> 2022.
> >
> > Based on the hostname, it seems Voilà's Z39.50 server uses
> > FirstSearch Z39.50, not WorldCat Z39.50, and is limited to records
> > and holdings for members of Canada's national union catalogue.
> > Either way, any characters outside of the ASCII range are corrupted, and LDR[09] has been reset to '
> > '.
> >
> > - If Voilà's Z39.50 server is a view of FirstSearch Z39.50, then OCLC
> > added a layer to strip the OCLC fields which introduces corruption.
> > - If it is a copy of a subset of WorldCat, then during the
> > dump/transform/load process, OCLC strips the OCLC fields in a way that
> > introduces corruption.
> >
> > I suppose I should report this problem to Library & Archives Canada,
> > as they are OCLC's direct customer and hopefully have some clout. I
> > do
> wonder,
> > though, if other national union catalogues that rely on OCLC are
> suffering
> > the same issues with their Z39.50 servers.
> >
> > Best,
> > Dan.
> >
> > On Wed, Nov 2, 2022 at 2:49 PM Haschart, Robert J (rh9ec) <
> > [log in to unmask]> wrote:
> >
> > > Dan,
> > >
> > > I have seen something similar recently. We (at the University of
> > > Virginia) were starting to see records containing the character. I
> > > wrote a simple script to take a list of records containing that
> > > character, and extract the OCLC number, and then programatically
> perform
> > a look-up for
> > > that OCLC number in WorldCAT via z39.50 The script would then
> compare
> > > our record with the one retrieved from OCLC and replace the fields
> > > containing the character with the corresponding field from the
> > > record from OCLC.
> > >
> > > For the most part this was working, but in several cases the record
> > > retrieved from OCLC contained the character. It is not merely an
> > > encoding/translation error since multiple different characters
> > > have
> been
> > > replaced with that character. Moreover, in some of the records
> returned
> > > there are some fields that contain correct accent marks while
> > > other fields that seemingly ought to contain the same accent mark, the
> > > accent mark has the character substituted in instead.
> > >
> > > One example shown below is the record for OCLC number 682540414
> > > (ISBN 9782760521292)
> > >
> > > Additionally, I used both my marc4j- based z39.50 libraries and
> > > the local copy of yaz-client to retrieve the exact record you used
> > > as an
> > example.
> > > The binary form of the record was identical for both methods of
> > retrieving
> > > the record. And there were no instances of the character in the
> > display
> > > of the record. (shown even further below)
> > >
> > >
> > > --Robert Haschart
> > > University of Virginia Library
> > >
> > >
> > > LEADER 03689cam a2200769Mi 4500
> > > 001 ocn682540414
> > > 003 OCoLC
> > > 005 20221102142039.0
> > > 008 101122s1996 quc o 000 0 fre d
> > > 006 m o d
> > > 007 cr |n|---|||||
> > > 040 $aEBLCP$beng$epn$cEBLCP$dOCLCQ$dAUW$dOCLCO$dP@U
> > >
> >
> $dOCLCF$dOCLCO$dOCLCQ$dCNKUC$dFVL$dOCLCQ$dOCLCA$dOCLCQ$dOCLCO$dOCLCQ$d
> JSTOR$dOCLCQ$dOCLCO$dQGK$dOCLCO$dOCL
> > > 066 $cZsym
> > > 016 $z96940719X
> > > 019 $a1259108319
> > > 020 $a9782760521292
> > > 020 $a276052129X
> > > 020 $a1441600825
> > > 020 $a9781441600820
> > > 035 $a(OCoLC)682540414$z(OCoLC)1259108319
> > > 037 $a22573/ctv18n0fbx$bJSTOR
> > > 050 4$aHD938
> > > 072 7$aBUS$x043000$2bisacsh
> > > 082 04$a338.4/791
> > > 100 1 $aLaplante, Marc.
> > > 245 12$aL'expérience touristique contemporaine :$bFondements
> > > sociaux et culturels.
> > > 260 $aQuébec :$bPresses de l''niversite du Quebec,$c1996.
> > > 300 $a1 online resource (204 pages)
> > > 336 $atext$btxt$2rdacontent
> > > 337 $acomputer$bc$2rdamedia
> > > 338 $aonline resource$bcr$2rdacarrier
> > > 490 1 $aCollection Tourisme
> > > 505 0 $aTable des matières; Liste des tableaux; Liste des
> > > figures; Introduction; Partie 1_L'enracinement du tourisme dans la
> > > socioté moderne; Chapitre 1_Touriste et tourisme d'hier et
> > > d'aujourd'hui; Chapitre 2_Sociétés et cultures de l'Occident
> > > industrialisé; Partie 2_Les dimensions de la culture touristique
> > > actuelle; Chapitre 3_L'expérience touristique comme expérience
> > > culturelle; Chapitre 4_La culture touristique personnelle;
> > > Chapitre 5_La culture touristique institutionnalisée; Conclusion; Guide bibliographique.
> > > 520 $aPour comprendre le tourisme comme fait social et culturel
> typique
> > > des sociétés postindustrielles, ce livre présente d'abord les
> > > principaux rapports entre tourisme et société puis élabore le
> > > concept de culture touristique.
> > > 588 0 $aPrint version record.
> > > 504 $aComprend des réf. bibliogr.
> > > 546 $aFrench.
> > > 650 0$aTourism and the arts.
> > > 650 0$aTourism$xSocial aspects.
> > > 650 0$aTourism.
> > > 650 0$aCulture.
> > > 650 0$aHeritage tourism.
> > > 650 6$aCulture$xAspect sociologique.
> > > 650 6$aTourisme culturel.
> > > 650 6$aTourisme$xAspect social.
> > > 650 6$aTourisme.
> > > 650 6$aTourisme et arts.
> > > 650 7$acultural tourism.$2aat
> > > 650 7$atourism.$2aat
> > > 650 7$aBUSINESS & ECONOMICS$xMarketing$xGeneral.$2bisacsh
> > > 650 7$aHeritage tourism.$2fast$0(OCoLC)fst00955465
> > > 650 7$aCulture.$2fast$0(OCoLC)fst00885059
> > > 650 7$aTourism.$2fast$0(OCoLC)fst01153142
> > > 650 7$aTourism and the arts.$2fast$0(OCoLC)fst01153221
> > > 650 7$aTourism$xSocial aspects.$2fast$0(OCoLC)fst01153202
> > > 655 4$aElectronic books.
> > > 776 08$iPrint version:$aLaplante, Marc.$tExpérience touristique
> > > contemporaine : Fondements sociaux et culturels.$dQuébec :
> > > Presses de l''niversite du Quebec, ©1996$z9782760509108
> > > 830 0$aTourisme (Presses de l'Université du Québec)
> > > 856
> > > 40$3desLibris$uhttps://nam10.safelinks.protection.outlook.com/?url
> > > =https%3A%2F%2Fwww.deslibris.ca%2FID%2F422431&data=05%7C01%7Cjaime
> > > taylor%40umass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33
> > > 954dc194bbd0b2e56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTW
> > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > VCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZDkgAPIFLePvtEdUSPErujWnqYNPq6Ps3
> > > cHFuDSmwu4%3D&reserved=0
> > > 856 40$3Project
> > > MUSE$uhttps://nam10.safelinks.protection.outlook.com/?url=https%3A
> > > %2F%2Fmuse.jhu.edu%2Fbooks%2F9782760521292&data=05%7C01%7Cjaimetay
> > > lor%40umass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33954
> > > dc194bbd0b2e56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTWFpb
> > > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > > 6Mn0%3D%7C3000%7C%7C%7C&sdata=9bZcBwBRIOZ76mA1aXNY4ILIpF1ozw00kbda
> > > PZizNOE%3D&reserved=0
> > > 856 40$3ProQuest Ebook Central$uhttps://
> > > public.ebookcentral.proquest.com/choice/publicfullrecord.aspx?p=60
> > > 0242
> > > 856
> > > 40$3JSTOR$uhttps://nam10.safelinks.protection.outlook.com/?url=htt
> > > ps%3A%2F%2Fwww.jstor.org%2Fstable%2F10.2307%2Fj.ctv18pgzcc&data=05
> > > %7C01%7Cjaimetaylor%40umass.edu%7Cff330dcb923244d6238108db20ce69c9
> > > %7C7bd08b0b33954dc194bbd0b2e56a497f%7C0%7C0%7C638139843154471347%7
> > > CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI
> > > 6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BqrcUi3TiHXj%2BaLu
> > > 5njwwfjP%2F7FyV57wmsz9wXCFB14%3D&reserved=0
> > > 880 0$6830-00/Zsym$aCollection Tourisme (Presses de l'Universit e
> > > du Qu ebec)
> > > 938 $aProQuest Ebook Central$bEBLB$nEBL600242
> > > 938 $aProject MUSE$bMUSE$nmuse20945
> > > 994 $aZ0$bVA@
> > > 948 $hHELD BY VA@ - 86 OTHER HOLDINGS
> > >
> > > LEADER 02823cam a2200721 a 4500
> > > 001 ocn276644602
> > > 003 OCoLC
> > > 005 20221102144112.0
> > > 008 081121s2008 quca b 000 p eng
> > > 040
> > >
> >
> $aNLC$beng$cNLC$dCDX$dBWX$dSUC$dUKMGB$dOCLCF$dNLC$dOCLCA$dREB$dOCLCA$d
> OCLCQ$dOCLCO
> > > 015 $a20089424808E$2can
> > > 015 $a20089424808$2can
> > > 016 $a(AMICUS)000034543125
> > > 016 7 $a015299625$2Uk
> > > 020 $a9782920325241$q(Galerie de l'UQAM)
> > > 020 $a2920325248
> > > 020 $a9782980985249$q(Galerie Simon Blais)
> > > 020 $a2980985244
> > > 035 $a(OCoLC)276644602
> > > 041 0 $aeng$afre
> > > 043 $an-cn-qu
> > > 050 4$aN6549.G686$bA4 2008
> > > 055 02$aNB249*
> > > 055 3$aN6549 G686$bA4 2009
> > > 082 04$a730.92$222
> > > 084 $acci1icc$2lacc
> > > 100 1 $aDéry, Louise,$d1955-
> > > 245 10$aRêver le nouveau monde :$bune œuvre d'art public de
> > > Michel Goulet offerte par la ville de Montréal à la ville de
> > > Québec dans le cadre des célébrations du 400e anniversaire de
> > > sa fondation /$ctextes, Louise Déry et Michel Goulet.
> > > 246 1 $iAdded title page in English:$aDreaming a new world
> > > 260 $a[Montréal] :$bGalerie Simon Blais,$c2008.
> > > 300 $a69 pages :$billustrations (some color) ;$c20 cm
> > > 336 $atext$btxt$2rdacontent
> > > 337 $aunmediated$bn$2rdamedia
> > > 338 $avolume$bnc$2rdacarrier
> > > 500 $aIncludes poems.
> > > 500 $aTranslation: Donald Pistolesi.
> > > 504 $aIncludes bibliographical references (pages 68-69).
> > > 546 $aText in French and English.
> > > 550 $aCo-published by: City of Montréal and Galerie de l'UQAM.
> > > 600 10$aGoulet, Michel,$d1944-
> > > 650 0$aPublic art$zQuébec (Province)$zQuébec.
> > > 650 0$aSculpture, French-Canadian$zQuébec (Province)$y21st century.
> > > 650 5$aSculpture, French Canadian$zQuébec (Province)$y21st century.
> > > 650 6$aSculpture québécoise$y21e siècle.
> > > 600 17$aGoulet, Michel,$d1944-$2fast$0(OCoLC)fst00070599
> > > 650 7$aPublic art.$2fast$0(OCoLC)fst01082064
> > > 650 7$aSculpture, French-Canadian.$2fast$0(OCoLC)fst01895320
> > > 651 7$aQuébec.$2fast$0(OCoLC)fst01207316
> > > 651 7$aQuébec$zQuébec.$2fast$0(OCoLC)fst01206798
> > > 648 7$a2000-2099$2fast
> > > 700 1 $aGoulet, Michel,$d1944-
> > > 710 2 $aGalerie de l'UQAM.
> > > 710 2 $aGalerie Simon Blais.
> > > 710 1 $aMontréal (Québec)
> > > 856 42$3French equivalent / Équivalent français$uhttps://
> > > bac-lac.on.worldcat.org/oclc/276644605
> > > 938 $aCoutts Information Services$bCOUT$n9265731
> > > 029 0 $aNLC$b000034543125
> > > 029 0 $aNLC$b20089424808E
> > > 029 1 $aBWX$bR3781137
> > > 029 1 $aCDX$b9265731
> > > 029 1 $aUKMGB$b015299625
> > > 994 $aZ0$bVA@
> > > 948 $hNO HOLDINGS IN VA@ - 21 OTHER HOLDINGS
> > >
> > > ________________________________
> > > From: Code for Libraries <[log in to unmask]> on behalf of
> > > Dan Scott <[log in to unmask]>
> > > Sent: Tuesday, November 1, 2022 11:32 AM
> > > To: [log in to unmask] <[log in to unmask]>
> > > Subject: [CODE4LIB] Connecting to OCLC WorldCat Z39.50 servers:
> > > character encoding issues
> > >
> > > Hello:
> > >
> > > I'm wondering if anyone else has run into this situation with OCLC
> > > WorldCat
> > > Z39.50 servers. When we search Voilà (Canada's national union
> > > catalogue, built on WorldCat)
> > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flibrary-archives.canada.ca%2Feng%2Fcollection%2Fbasics%2Fpages%2Fvoila&data=05%7C01%7Cjaimetaylor%40umass.edu%7Cff330dcb923244d6238108db20ce69c9%7C7bd08b0b33954dc194bbd0b2e56a497f%7C0%7C0%7C638139843154471347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EyzmniJQLFhseYls%2B2bO770QKz587XFeCRTc9OX9vkM%3D&reserved=0.
> > > aspx
> > > >
> > > via Z39.50 for a record that is catalogued in French for ILL
> > > purposes, the resulting records contain corrupted characters where
> > > accented characters are expected. We're seeing this in Ex Libris
> > > Alma, where Voilà is an external search source maintained by Alma
> > > (we can't see the parameters Alma is using to connect).
> > >
> > > For example, searching Voilà for ISBN 9782920325241 returns a list
> > > of records for which the summaries display without accents.
> > > However, checking the MARC record itself shows that where there
> > > should be accents, there are instead literals in place of Unicode codepoints.
> > >
> > > The 245 for the first record returned for ISBN 9782920325241
> > > appears as follows in Alma:
> > > 245 1 0 $$a R<U+00fd>ever le nouveau monde : $$b une <U+00fd>uvre
> > > d'art public de Michel Goulet offerte par la ville de
> > > Montr<U+00fd>eal <U+00fd>a la ville de Qu<U+00fd>ebec dans le
> > > cadre des c<U+00fd>el<U+00fd>ebrations du 400e anniversaire de sa
> > > fondation / $$c textes, Louise D<U+00fd>ery et Michel Goulet.
> > >
> > > - Where we see R<U+00fd>ever, we should see Rèver.
> > > - Where we see <U+00fd>uvre, we should see œuvre.
> > > - Where we see Montr<U+00fd>eal, we should see Montréal.
> > > - Where we see <U+00fd>a, we should see à.
> > >
> > > I don't *think *this is an example of a Unicode canonicalization
> > > into a normalized form using combining characters, as <U+00fd> is
> > > not a combining character. If that was the case, we would see
> > > <U+0301>e for é, <U+0300>a for à, etc.
> > >
> > > Instead, it looks like a case of corruption introduced by
> > > incorrect decoding. This can occur when the Z39.50 source uses
> > > UTF8 and the client interprets it as MARC8, or vice versa.
> > >
> > > That said, I have tested a few different approaches to querying
> > > Voilà using the yaz-client command line Z39.50 client (version
> > > 5.28.0 on Ubuntu), and it seems that Voilà itself might be
> > > returning the
> corrupted
> > characters.
> > >
> > > For example, when I set the negotiation character set to MARC-8
> > > (matching the LDR[09] of the returned record) and the display
> > > character set to UTF-8, the results on the screen and in the file
> > > show
> > corruption:
> > >
> > > $ yaz-client -m testvoila.mrc
> > > Z> negcharset MARC-8
> > > Character set negotiation : MARC-8
> > > Z> charset
> > > Negotiation character set `MARC-8'
> > > Records in charset yes
> > > Charneg version 3
> > > Display character set is `UTF-8'
> > > MARC character set is `none'
> > > Query character set is `none'
> > > Z> authentication USERNAME/PASSWORD
> > > Authentication set to Open (USERNAME/PASSWORD)
> > > Z> open fsz3950.oclc.org:210/gclacConnecting...OK.
> > > Sent initrequest.
> > > Connection accepted by v3 target.
> > > UserInformationfield:
> > > {
> > > OID: 1 2 840 10003 10 1000 17 1
> > > {
> > > ANY (len=66)
> > > }
> > > }
> > > OCLC UserInformation:
> > > {
> > > motd 'You are searching the FirstSearch 5.0 Z39.50 Server!'
> > > dblist {
> > > 'laccat'
> > > }
> > > }
> > > Options: search present delSet triggerResourceCtrl scan sort
> > > extendedServices namedResultSets
> > > Elapsed: 0.167063
> > > Z> find @attr 1=7 9782920325241
> > > Sent searchRequest.
> > > Received SearchResponse.
> > > Search was a success.
> > > Number of hits: 6, setno 1
> > > records returned: 0
> > > Elapsed: 0.190875
> > > Z> show
> > > Sent presentRequest (1+1). Records: 1 [laccat] Record type: USmarc
> > > 02802cam 2200685 a 4500 008 20081121s2008 quca b 000 p eng
> > > 015 $a 20089424808E $2 can
> > > 015 $a 20089424808 $2 can
> > > 016 $a (AMICUS)000034543125
> > > 016 7 $a 015299625 $2 Uk
> > > 020 $a 9782920325241 $q (Galerie de l'UQAM)
> > > 020 $a 2920325248
> > > 020 $a 9782980985249 $q (Galerie Simon Blais)
> > > 020 $a 2980985244
> > > 040 $a NLC $b eng $c NLC $d CDX $d BWX $d SUC $d UKMGB $d OCLCF $d
> > > NLC $d OCLCA $d REB $d OCLCA $d OCLCQ $d OCLCO
> > > 041 0 $a eng $a fre
> > > 043 $a n-cn-qu 050 4 $a N6549.G686 $b A4 2008
> > > 055 02 $a NB249*
> > > 055 3 $a N6549 G686 $b A4 2009
> > > 082 04 $a 730.92 $2 22
> > > 084 $a cci1icc $2 lacc
> > > 100 1 $a D ery, Louise, $d 1955- 245 10 $a R ever le nouveau monde :
> > > $b une uvre d'art public de Michel Goulet offerte par la ville de
> > > Montr eal a la ville de Qu ebec dans le cadre des c el ebrations
> > > du 400e anniversaire de sa fondation / $c textes, Louise D ery et
> > > Michel
> > Goulet.
> > >
> > > In the file with the downloaded record (looking at it with Vim in
> > > binary
> > > mode) you can see the same <fd> characters as we see in Alma for
> > > the same
> > > record:
> > >
> > > 02802cam 2200685 a
> > >
> > > 450000800430000001500220004301500210006501600250008601600180011102
> > > 0003
> > > 900129020001500168020004100183020001500224040008500239041001300324
> > > 0430
> > > 012003370500024003490550011003730550024003840820015004080840018004
> > > 2310
> > > 000260044124502400046724600550070726000470076230000520080933600260
> > > 0861
> > > 337002800887338002700915500002000942500003500962504005500997546003
> > > 2010
> > > 525500062010846000027011466000033011736480020012066500045012266500
> > > 0660
> > > 127165000660133765000410140365000220144465000380146665100130150465
> > > 1002
> > > 201517700002701539710002301566710002501589710002401614856009401638
> > > 9380
> > > 047017329450008017799470030017879480141018179480038019589480047019
> > > 9694
> > > 8002902043948003202072949001202104^^20081121s2008
> > > quca b 000 p eng ^^ ^_a20089424808E^_2can^^ ^_a20089424808^_2can^^
> > > ^_a(AMICUS)000034543125^^7 ^_a015299625^_2Uk^^
> > > ^_a9782920325241^_q(Galerie de l'UQAM)^^ ^_a2920325248^^
> > > ^_a9782980985249^_q(Galerie Simon Blais)^^ ^_a2980985244^^
> > >
> > > ^_aNLC^_beng^_cNLC^_dCDX^_dBWX^_dSUC^_dUKMGB^_dOCLCF^_dNLC^_dOCLCA
> > > ^_dR
> > > EB^_dOCLCA^_dOCLCQ^_dOCLCO^^0 ^_aeng^_afre^^ ^_an-cn-qu^^
> > > 4^_aN6549.G686^_bA4 2008^^02^_aNB249*^^
> > > 3^_aN6549 G686^_bA4 2009^^04^_a730.92^_222^^ ^_acci1icc^_2lacc^^1
> > > ^_aD<fd>ery, Louise,^_d1955-^^10^_aR<fd>ever le nouveau monde
> > > :^_bune <fd>uvre d'art public de Michel Goulet offerte par la
> > > ville de Montr<fd>eal <fd>a la ville de Qu<fd>ebec dans le cadre
> > > des c<fd>el<fd>ebrations du 400e anniversaire de sa fondation
> > > /^_ctextes,
> > Louise D<fd>ery et Michel Goulet.
> > >
> > > If anyone has run into this with OCLC WorldCat Z39.50 and figured
> > > out a solution, I would love to hear it. It would even be helpful
> > > to know if you have a connection to an OCLC WorldCat Z39.50 server
> > > and aren't seeing these kinds of problems. I know most of you
> > > probably won't have authentication credentials for Voilà, but the
> > > setup has to be the same for other OCLC WorldCat Z39.50 servers... right?
> > >
> > > In Z39.50 and character encoding madness, I remain, Dan
> > >
> >
>
