LISTSERV mailing list manager LISTSERV 16.5

Help for CODE4LIB Archives


CODE4LIB Archives

CODE4LIB Archives


CODE4LIB@LISTS.CLIR.ORG


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CODE4LIB Home

CODE4LIB Home

CODE4LIB  November 2022

CODE4LIB November 2022

Subject:

Re: Connecting to OCLC WorldCat Z39.50 servers: character encoding issues

From:

Dan Scott <[log in to unmask]>

Reply-To:

Code for Libraries <[log in to unmask]>

Date:

Wed, 2 Nov 2022 16:08:19 -0400

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (372 lines)

Robert, thanks so much for replying with your findings! Super helpful.

Based on the record you shared, I noted a few differences between what I'm
seeing with Voilà and "real" WorldCat:

   - Voilà does not include OCLC numbers; the 001, 003, and 035 are
   stripped (likely other fields as well)
   - Voilà returns records encoded in MARC8 whereas WorldCat returns
   records encoded in UTF8

I tried more searches against Voilà's Z39.50 server. In every case where
there should be an accented character, the same corruption occurs.

I also noticed that WorldCat's Z39.50 server uses the hostname zcat.oclc.org
and is documented quite differently
<https://help.oclc.org/Metadata_Services/Z3950_Cataloging/Get_started/Configuration_guide_for_OCLC_Z39.50_Cataloging>
from FirstSearch's Z39.50 service, which uses the hostname fsz3950.oclc.org.
The documentation for FirstSearch Z39.50
<https://help.oclc.org/Discovery_and_Reference/FirstSearch/Z3950_access/Z3950_configuration_for_FirstSearch_databases>
doesn't even mention encodings, which seems like quite a red flag in 2022.

Based on the hostname, it seems Voilà's Z39.50 server uses FirstSearch
Z39.50, not WorldCat Z39.50, and is limited to records and holdings for
members of Canada's national union catalogue. Either way, any characters
outside of the ASCII range are corrupted, and LDR[09] has been reset to '
'.

   - If Voilà's Z39.50 server is a view of FirstSearch Z39.50, then OCLC
   added a layer to strip the OCLC fields which introduces corruption.
   - If it is a copy of a subset of WorldCat, then during the
   dump/transform/load process, OCLC strips the OCLC fields in a way that
   introduces corruption.

I suppose I should report this problem to Library & Archives Canada, as
they are OCLC's direct customer and hopefully have some clout. I do wonder,
though, if other national union catalogues that rely on OCLC are suffering
the same issues with their Z39.50 servers.

Best,
Dan.

On Wed, Nov 2, 2022 at 2:49 PM Haschart, Robert J (rh9ec) <
[log in to unmask]> wrote:

> Dan,
>
> I have seen something similar recently.   We (at the University of
> Virginia) were starting to see records containing the � character.​   I
> wrote a simple script to take a list of records containing that character,
> and extract the OCLC number, and then programatically perform a look-up for
> that OCLC number in WorldCAT via z39.50    The script would then compare
> our record with the one retrieved from OCLC and replace the fields
> containing the � character with the corresponding field from the record
> from OCLC.
>
> For the most part this was working, but in several cases the record
> retrieved from OCLC contained the � character.    It is not merely an
> encoding/translation error since multiple different characters have been
> replaced with that character.   Moreover, in some of the records returned
> there are some fields that contain correct accent marks while other fields
> that seemingly ought to contain the same accent mark, the accent mark has
> the � character substituted in instead.
>
> One example shown below is the record for OCLC number     682540414
> (ISBN   9782760521292)
>
> Additionally, I used both my marc4j- based z39.50  libraries and the local
> copy of yaz-client to retrieve the exact record you used as an example.
>  The binary form of the record was identical for both methods of retrieving
> the record.   And there were no instances of the � character in the display
> of the record.   (shown even further below)
>
>
> --Robert Haschart
> University of Virginia Library
>
>
> LEADER 03689cam a2200769Mi 4500
> 001 ocn682540414
> 003 OCoLC
> 005 20221102142039.0
> 008 101122s1996    quc     o     000 0 fre d
> 006 m     o  d
> 007 cr |n|---|||||
> 040   $aEBLCP$beng$epn$cEBLCP$dOCLCQ$dAUW$dOCLCO$dP@U
> $dOCLCF$dOCLCO$dOCLCQ$dCNKUC$dFVL$dOCLCQ$dOCLCA$dOCLCQ$dOCLCO$dOCLCQ$dJSTOR$dOCLCQ$dOCLCO$dQGK$dOCLCO$dOCL
> 066   $cZsym
> 016   $z96940719X
> 019   $a1259108319
> 020   $a9782760521292
> 020   $a276052129X
> 020   $a1441600825
> 020   $a9781441600820
> 035   $a(OCoLC)682540414$z(OCoLC)1259108319
> 037   $a22573/ctv18n0fbx$bJSTOR
> 050  4$aHD938
> 072  7$aBUS$x043000$2bisacsh
> 082 04$a338.4/791
> 100 1 $aLaplante, Marc.
> 245 12$aL'expérience touristique contemporaine :$bFondements sociaux et
> culturels.
> 260   $aQuébec :$bPresses de l''niversite du Quebec,$c1996.
> 300   $a1 online resource (204 pages)
> 336   $atext$btxt$2rdacontent
> 337   $acomputer$bc$2rdamedia
> 338   $aonline resource$bcr$2rdacarrier
> 490 1 $aCollection Tourisme
> 505 0 $aTable des matières; Liste des tableaux; Liste des figures;
> Introduction; Partie 1_L'enracinement du tourisme dans la socioté moderne;
> Chapitre 1_Touriste et tourisme d'hier et d'aujourd'hui; Chapitre
> 2_Sociétés et cultures de l'Occident industrialisé; Partie 2_Les
> dimensions de la culture touristique actuelle; Chapitre 3_L'expérience
> touristique comme expérience culturelle; Chapitre 4_La culture touristique
> personnelle; Chapitre 5_La culture touristique institutionnalisée;
> Conclusion; Guide bibliographique.
> 520   $aPour comprendre le tourisme comme fait social et culturel typique
> des sociétés postindustrielles, ce livre présente d'abord les principaux
> rapports entre tourisme et société puis élabore le concept de culture
> touristique.
> 588 0 $aPrint version record.
> 504   $aComprend des réf. bibliogr.
> 546   $aFrench.
> 650  0$aTourism and the arts.
> 650  0$aTourism$xSocial aspects.
> 650  0$aTourism.
> 650  0$aCulture.
> 650  0$aHeritage tourism.
> 650  6$aCulture$xAspect sociologique.
> 650  6$aTourisme culturel.
> 650  6$aTourisme$xAspect social.
> 650  6$aTourisme.
> 650  6$aTourisme et arts.
> 650  7$acultural tourism.$2aat
> 650  7$atourism.$2aat
> 650  7$aBUSINESS & ECONOMICS$xMarketing$xGeneral.$2bisacsh
> 650  7$aHeritage tourism.$2fast$0(OCoLC)fst00955465
> 650  7$aCulture.$2fast$0(OCoLC)fst00885059
> 650  7$aTourism.$2fast$0(OCoLC)fst01153142
> 650  7$aTourism and the arts.$2fast$0(OCoLC)fst01153221
> 650  7$aTourism$xSocial aspects.$2fast$0(OCoLC)fst01153202
> 655  4$aElectronic books.
> 776 08$iPrint version:$aLaplante, Marc.$tExpérience touristique
> contemporaine : Fondements sociaux et culturels.$dQuébec : Presses de
> l''niversite du Quebec, ©1996$z9782760509108
> 830  0$aTourisme (Presses de l'Université du Québec)
> 856 40$3desLibris$uhttps://www.deslibris.ca/ID/422431
> 856 40$3Project MUSE$uhttps://muse.jhu.edu/books/9782760521292
> 856 40$3ProQuest Ebook Central$uhttps://
> public.ebookcentral.proquest.com/choice/publicfullrecord.aspx?p=600242
> 856 40$3JSTOR$uhttps://www.jstor.org/stable/10.2307/j.ctv18pgzcc
> 880  0$6830-00/Zsym$aCollection Tourisme (Presses de l'Universit�e du
> Qu�ebec)
> 938   $aProQuest Ebook Central$bEBLB$nEBL600242
> 938   $aProject MUSE$bMUSE$nmuse20945
> 994   $aZ0$bVA@
> 948   $hHELD BY VA@ - 86 OTHER HOLDINGS
>
> LEADER 02823cam a2200721 a 4500
> 001 ocn276644602
> 003 OCoLC
> 005 20221102144112.0
> 008 081121s2008    quca     b    000 p eng
> 040
>  $aNLC$beng$cNLC$dCDX$dBWX$dSUC$dUKMGB$dOCLCF$dNLC$dOCLCA$dREB$dOCLCA$dOCLCQ$dOCLCO
> 015   $a20089424808E$2can
> 015   $a20089424808$2can
> 016   $a(AMICUS)000034543125
> 016 7 $a015299625$2Uk
> 020   $a9782920325241$q(Galerie de l'UQAM)
> 020   $a2920325248
> 020   $a9782980985249$q(Galerie Simon Blais)
> 020   $a2980985244
> 035   $a(OCoLC)276644602
> 041 0 $aeng$afre
> 043   $an-cn-qu
> 050  4$aN6549.G686$bA4 2008
> 055 02$aNB249*
> 055  3$aN6549 G686$bA4 2009
> 082 04$a730.92$222
> 084   $acci1icc$2lacc
> 100 1 $aDéry, Louise,$d1955-
> 245 10$aRêver le nouveau monde :$bune œuvre d'art public de Michel Goulet
> offerte par la ville de Montréal à la ville de Québec dans le cadre des
> célébrations du 400e anniversaire de sa fondation /$ctextes, Louise Déry
> et Michel Goulet.
> 246 1 $iAdded title page in English:$aDreaming a new world
> 260   $a[Montréal] :$bGalerie Simon Blais,$c2008.
> 300   $a69 pages :$billustrations (some color) ;$c20 cm
> 336   $atext$btxt$2rdacontent
> 337   $aunmediated$bn$2rdamedia
> 338   $avolume$bnc$2rdacarrier
> 500   $aIncludes poems.
> 500   $aTranslation: Donald Pistolesi.
> 504   $aIncludes bibliographical references (pages 68-69).
> 546   $aText in French and English.
> 550   $aCo-published by: City of Montréal and Galerie de l'UQAM.
> 600 10$aGoulet, Michel,$d1944-
> 650  0$aPublic art$zQuébec (Province)$zQuébec.
> 650  0$aSculpture, French-Canadian$zQuébec (Province)$y21st century.
> 650  5$aSculpture, French Canadian$zQuébec (Province)$y21st century.
> 650  6$aSculpture québécoise$y21e siècle.
> 600 17$aGoulet, Michel,$d1944-$2fast$0(OCoLC)fst00070599
> 650  7$aPublic art.$2fast$0(OCoLC)fst01082064
> 650  7$aSculpture, French-Canadian.$2fast$0(OCoLC)fst01895320
> 651  7$aQuébec.$2fast$0(OCoLC)fst01207316
> 651  7$aQuébec$zQuébec.$2fast$0(OCoLC)fst01206798
> 648  7$a2000-2099$2fast
> 700 1 $aGoulet, Michel,$d1944-
> 710 2 $aGalerie de l'UQAM.
> 710 2 $aGalerie Simon Blais.
> 710 1 $aMontréal (Québec)
> 856 42$3French equivalent / Équivalent français$uhttps://
> bac-lac.on.worldcat.org/oclc/276644605
> 938   $aCoutts Information Services$bCOUT$n9265731
> 029 0 $aNLC$b000034543125
> 029 0 $aNLC$b20089424808E
> 029 1 $aBWX$bR3781137
> 029 1 $aCDX$b9265731
> 029 1 $aUKMGB$b015299625
> 994   $aZ0$bVA@
> 948   $hNO HOLDINGS IN VA@ - 21 OTHER HOLDINGS
>
> ________________________________
> From: Code for Libraries <[log in to unmask]> on behalf of Dan Scott
> <[log in to unmask]>
> Sent: Tuesday, November 1, 2022 11:32 AM
> To: [log in to unmask] <[log in to unmask]>
> Subject: [CODE4LIB] Connecting to OCLC WorldCat Z39.50 servers: character
> encoding issues
>
> Hello:
>
> I'm wondering if anyone else has run into this situation with OCLC WorldCat
> Z39.50 servers. When we search Voilà (Canada's national union catalogue,
> built on WorldCat)
> <https://library-archives.canada.ca/eng/collection/basics/pages/voila.aspx
> >
> via Z39.50 for a record that is catalogued in French for ILL purposes, the
> resulting records contain corrupted characters where accented characters
> are expected. We're seeing this in Ex Libris Alma, where Voilà is an
> external search source maintained by Alma (we can't see the parameters Alma
> is using to connect).
>
> For example, searching Voilà for ISBN 9782920325241 returns a list of
> records for which the summaries display without accents. However, checking
> the MARC record itself shows that where there should be accents, there are
> instead literals in place of Unicode codepoints.
>
> The 245 for the first record returned for ISBN 9782920325241 appears as
> follows in Alma:
> 245 1 0 $$a R<U+00fd>ever le nouveau monde : $$b une <U+00fd>uvre d'art
> public de Michel Goulet offerte par la ville de Montr<U+00fd>eal <U+00fd>a
> la ville de Qu<U+00fd>ebec dans le cadre des c<U+00fd>el<U+00fd>ebrations
> du 400e anniversaire de sa fondation / $$c textes, Louise D<U+00fd>ery et
> Michel Goulet.
>
>    - Where we see R<U+00fd>ever, we should see Rèver.
>    - Where we see <U+00fd>uvre, we should see œuvre.
>    - Where we see Montr<U+00fd>eal, we should see Montréal.
>    - Where we see <U+00fd>a, we should see à.
>
> I don't *think *this is an example of a Unicode canonicalization into a
> normalized form using combining characters, as <U+00fd> is not a combining
> character. If that was the case, we would see <U+0301>e for é, <U+0300>a
> for à, etc.
>
> Instead, it looks like a case of corruption introduced by incorrect
> decoding. This can occur when the Z39.50 source uses UTF8 and the client
> interprets it as MARC8, or vice versa.
>
> That said, I have tested a few different approaches to querying Voilà using
> the yaz-client command line Z39.50 client (version 5.28.0 on Ubuntu), and
> it seems that Voilà itself might be returning the corrupted characters.
>
> For example, when I set the negotiation character set to MARC-8 (matching
> the LDR[09] of the returned record) and the display character set to UTF-8,
> the results on the screen and in the file show corruption:
>
> $ yaz-client -m testvoila.mrc
> Z> negcharset MARC-8
> Character set negotiation : MARC-8
> Z> charset
> Negotiation character set `MARC-8'
> Records in charset yes
> Charneg version 3
> Display character set is `UTF-8'
> MARC character set is `none'
> Query character set is `none'
> Z> authentication USERNAME/PASSWORD
> Authentication set to Open (USERNAME/PASSWORD)
> Z> open fsz3950.oclc.org:210/gclacConnecting...OK.
> Sent initrequest.
> Connection accepted by v3 target.
> UserInformationfield:
> {
>   OID: 1 2 840 10003 10 1000 17 1
>   {
>     ANY (len=66)
>   }
> }
> OCLC UserInformation:
> {
>   motd 'You are searching the FirstSearch 5.0 Z39.50 Server!'
>   dblist {
>     'laccat'
>   }
> }
> Options: search present delSet triggerResourceCtrl scan sort
> extendedServices namedResultSets
> Elapsed: 0.167063
> Z> find @attr 1=7 9782920325241
> Sent searchRequest.
> Received SearchResponse.
> Search was a success.
> Number of hits: 6, setno 1
> records returned: 0
> Elapsed: 0.190875
> Z> show
> Sent presentRequest (1+1). Records: 1 [laccat]
> Record type: USmarc
> 02802cam 2200685 a 4500 008 20081121s2008 quca b 000 p eng
> 015 $a 20089424808E $2 can
> 015 $a 20089424808 $2 can
> 016 $a (AMICUS)000034543125
> 016 7 $a 015299625 $2 Uk
> 020 $a 9782920325241 $q (Galerie de l'UQAM)
> 020 $a 2920325248
> 020 $a 9782980985249 $q (Galerie Simon Blais)
> 020 $a 2980985244
> 040 $a NLC $b eng $c NLC $d CDX $d BWX $d SUC $d UKMGB $d OCLCF $d NLC $d
> OCLCA $d REB $d OCLCA $d OCLCQ $d OCLCO
> 041 0 $a eng $a fre
> 043 $a n-cn-qu 050 4 $a N6549.G686 $b A4 2008
> 055 02 $a NB249*
> 055 3 $a N6549 G686 $b A4 2009
> 082 04 $a 730.92 $2 22
> 084 $a cci1icc $2 lacc
> 100 1 $a D�ery, Louise, $d 1955- 245 10 $a R�ever le nouveau monde : $b une
> �uvre d'art public de Michel Goulet offerte par la ville de Montr�eal �a la
> ville de Qu�ebec dans le cadre des c�el�ebrations du 400e anniversaire de
> sa fondation / $c textes, Louise D�ery et Michel Goulet.
>
> In the file with the downloaded record (looking at it with Vim in binary
> mode) you can see the same <fd> characters as we see in Alma for the same
> record:
>
> 02802cam 2200685 a
>
> 4500008004300000015002200043015002100065016002500086016001800111020003900129020001500168020004100183020001500224040008500239041001300324043001200337050002400349055001100373055002400384082001500408084001800423100002600441245024000467246005500707260004700762300005200809336002600861337002800887338002700915500002000942500003500962504005500997546003201052550006201084600002701146600003301173648002001206650004501226650006601271650006601337650004101403650002201444650003801466651001301504651002201517700002701539710002301566710002501589710002401614856009401638938004701732945000801779947003001787948014101817948003801958948004701996948002902043948003202072949001202104^^20081121s2008
> quca b 000 p eng ^^ ^_a20089424808E^_2can^^ ^_a20089424808^_2can^^
> ^_a(AMICUS)000034543125^^7 ^_a015299625^_2Uk^^ ^_a9782920325241^_q(Galerie
> de l'UQAM)^^ ^_a2920325248^^ ^_a9782980985249^_q(Galerie Simon Blais)^^
> ^_a2980985244^^
>
> ^_aNLC^_beng^_cNLC^_dCDX^_dBWX^_dSUC^_dUKMGB^_dOCLCF^_dNLC^_dOCLCA^_dREB^_dOCLCA^_dOCLCQ^_dOCLCO^^0
> ^_aeng^_afre^^ ^_an-cn-qu^^ 4^_aN6549.G686^_bA4 2008^^02^_aNB249*^^
> 3^_aN6549 G686^_bA4 2009^^04^_a730.92^_222^^ ^_acci1icc^_2lacc^^1
> ^_aD<fd>ery, Louise,^_d1955-^^10^_aR<fd>ever le nouveau monde :^_bune
> <fd>uvre d'art public de Michel Goulet offerte par la ville de Montr<fd>eal
> <fd>a la ville de Qu<fd>ebec dans le cadre des c<fd>el<fd>ebrations du 400e
> anniversaire de sa fondation /^_ctextes, Louise D<fd>ery et Michel Goulet.
>
> If anyone has run into this with OCLC WorldCat Z39.50 and figured out a
> solution, I would love to hear it. It would even be helpful to know if you
> have a connection to an OCLC WorldCat Z39.50 server and aren't seeing these
> kinds of problems. I know most of you probably won't have authentication
> credentials for Voilà, but the setup has to be the same for other OCLC
> WorldCat Z39.50 servers... right?
>
> In Z39.50 and character encoding madness, I remain,
> Dan
>

Top of Message | Previous Page | Permalink

Advanced Options


Options

Log In

Log In

Get Password

Get Password


Search Archives

Search Archives


Subscribe or Unsubscribe

Subscribe or Unsubscribe


Archives

March 2024
February 2024
January 2024
December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
September 2004
August 2004
July 2004
June 2004
May 2004
April 2004
March 2004
February 2004
January 2004
December 2003
November 2003

ATOM RSS1 RSS2



LISTS.CLIR.ORG

CataList Email List Search Powered by the LISTSERV Email List Manager