 |
 |
Binkley, Peter wrote: >Bear in mind that even in UTF-8 there is more than one way to encode an >accented character. It can be precomposed (using a single character, >e.g. U0089 for lower-case e-acute: this is normalization form C) or >decomposed (using a base character and a non-spacing diacritic, e.g. >U0065 and U0301, lower-case e plus the acute accent: this is >normalization form D). If you're searching at the byte level, you have >to be sure that your index and your search term have been normalized the >same way or they won't match. I've found this FAQ useful for this stuff: >http://pipin.tmd.ns.ac.yu/unicode/unicode-faq.html. In a Java context, >we've used ICU4J (http://icu.sourceforge.net) to normalize stuff >(including stripping accents and normalizing case for different scripts) >for indexing and searching in UTF-8. There's also a C API, which could >presumably be incorporated into a Perl process, but no doubt there are >similar native Perl tools. > >In general I think we've got to include i18n from the beginning: pay >attention to character sets of incoming data, normalize as early in the >process as possible (especially if ANSEL is involved!), use >UTF-8-compliant tools, and be consistent. Deliver UTF-8 to the browser >(this site helps with the html: >http://ppewww.ph.gla.ac.uk/~flavell/charset/quick.html). This is still >not as easy as it ought to be but at least there are good open-source >tools out there. > > Wow, it looks like there are some unicode experts at our midst. I am in the middle of developing an international bibliographic database where most of the titles are in languages other than EN-US. Our database will store citations entered in via a web form since the bibliography is in card format. I am using MySQL 4 because of the unicode support and collations. I normally use postgres, but I figured for a database that will mainly be used for searching only (very little writes after the data has been populated) i'd give MySQL a try. One feature we would like to offer is searching via the collations. For example, if I enter the phrase francais, i would hope that any items with the term français would result. Is it correct to use MySQL's collations for this? Does anyone have experience with this? I am still learning the uses of UTF-8 characters, so I am glad there are so many of you who know so much about this on this list! Andrew