 |
 |
Hi Eric, If you're pretty sure you indexed the characters properly and are getting garbage no matter what you do, my first thought is that this is a localization issue. Can you cat/grep/sed/vi/whatever these characters in a terminal window? If not, that is at least part of your problem. Running locale-gen en_US.UTF-8 may help. <rant>Why this hasn't been a default for many years is ridiculous, but I digress</rant> If you can type/read these characters in a terminal window, the problem is downstream. In that case, I'd verify the Accept-Encoding headers and keep working your way down until you hit the index. kyle On Tue, Aug 14, 2018 at 7:50 AM Eric Lease Morgan <[log in to unmask]> wrote: > How to I go about indexing & searching Chinese text using Solr? > > I have a pile o' simplified Chinese text encoded in UTF-8. Taking hints > from some Solr documentation [1], I have configured my index thusly: > > <schema name="hui" version="1.6"> > <uniqueKey>key</uniqueKey> > > <!-- local field types --> > <fieldType name="string" class="solr.StrField" sortMissingLast="true" > docValues="true"/> > <fieldType name="long" class="solr.TrieLongField" > positionIncrementGap="0" docValues="true" precisionStep="0"/> > > <!-- chinese indexing configuration happens here --> > <fieldType name="text_general" class="solr.TextField"> > <analyzer> > <tokenizer class="solr.HMMChineseTokenizerFactory"/> > <filter class="solr.CJKWidthFilterFactory"/> > <filter class="solr.StopFilterFactory" > words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/> > <filter class="solr.PorterStemFilterFactory"/> > <filter class="solr.LowerCaseFilterFactory"/> > </analyzer> > </fieldType> > > <!-- good to have fields --> > <field name="_root_" type="string" docValues="false" > indexed="true" stored="false"/> > <field name="_text_" type="text_general" multiValued="true" > indexed="true" stored="false"/> > <field name="_version_" type="long" indexed="true" > stored="false"/> > > <!-- my fields --> > <field name="fulltext" type="text_general" multiValued="false" > indexed="true" stored="true" /> > <field name="key" type="text_general" multiValued="false" > indexed="true" stored="true" /> > > <!-- cool field --> > <copyField source="*" dest="_text_"/> > > </schema> > > I then index my content, and as per the Solr Admin interface, my index > includes 130 documents. I then have problems searching: > > * Using the Solr Admin interface, I can search for everything (*:*), and > all my results are returned, but the Chinese characters are all mangled. > > * Using a Lynx (terminal) interface, I can search for everything (*:*), > and all my results are returned, but the Chinese characters are all mangled. > > * Using a Perl interface of my own design, I can search for everything > (*:*), and get results and the characters are NOT mangled. > > * Using the same Perl interface, I try to enter a query using Chinese > characters, but I always get zero results. > > * Using the same Perl interface, I can search for the word "body" (an > HTML element I didn't delete), I get expected results. > > * Using the same Perl interface, I an enter a query using the mangled > characters, and I get the sorts of results I expect. > > I believe I have indexed my documents "correctly", but I can't seem to > query the index in the expected manner. What might I be doing wrong? > > [1] Solr documentation - > https://lucene.apache.org/solr/guide/6_6/language-analysis.html#LanguageAnalysis-TraditionalChinese > > -- > Eric Lease Morgan > Digital Initiatives Librarian, Navari Family Center for Digital Scholarship > Hesburgh Libraries > > University of Notre Dame > 250E Hesburgh Library > Notre Dame, IN 46556 > o: 574-631-8604 > w: cds.library.nd.edu >