Eric -

How did you index the files?    These are, I presume (based on "body" mention below) these are HTML files?    Can you send along a file (direct to me is fine) and how you indexed it and I'll take a look.

	Erik


> On Aug 14, 2018, at 10:49 AM, Eric Lease Morgan <[log in to unmask]> wrote:
> 
> How to I go about indexing & searching Chinese text using Solr?
> 
> I have a pile o' simplified Chinese text encoded in UTF-8. Taking hints from some Solr documentation [1], I have configured my index thusly:
> 
>  <schema name="hui" version="1.6">
>    <uniqueKey>key</uniqueKey>
> 
>    <!-- local field types -->
>    <fieldType name="string" class="solr.StrField" sortMissingLast="true" docValues="true"/>
>    <fieldType name="long" class="solr.TrieLongField" positionIncrementGap="0" docValues="true" precisionStep="0"/>
> 
>    <!-- chinese indexing configuration happens here -->
>    <fieldType name="text_general" class="solr.TextField">
>      <analyzer>
>        <tokenizer class="solr.HMMChineseTokenizerFactory"/>
>        <filter class="solr.CJKWidthFilterFactory"/>
>        <filter class="solr.StopFilterFactory"
>          words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/>
>        <filter class="solr.PorterStemFilterFactory"/>
>        <filter class="solr.LowerCaseFilterFactory"/>
>      </analyzer>
>   </fieldType>
> 
>    <!-- good to have fields -->
>    <field name="_root_"    type="string"       docValues="false"   indexed="true" stored="false"/>
>    <field name="_text_"    type="text_general" multiValued="true"  indexed="true" stored="false"/>
>    <field name="_version_" type="long"         indexed="true"      stored="false"/>
> 
>    <!-- my fields -->
>    <field name="fulltext" type="text_general"  multiValued="false"  indexed="true"   stored="true" />
>    <field name="key"      type="text_general"  multiValued="false"  indexed="true"   stored="true"  />
> 
>    <!-- cool field -->
>    <copyField source="*" dest="_text_"/>
> 
>  </schema>
> 
> I then index my content, and as per the Solr Admin interface, my index includes 130 documents. I then have problems searching:
> 
>  * Using the Solr Admin interface, I can search for everything (*:*), and all my results are returned, but the Chinese characters are all mangled.
> 
>  * Using a Lynx (terminal) interface, I can search for everything (*:*), and all my results are returned, but the Chinese characters are all mangled.
> 
>  * Using a Perl interface of my own design, I can search for everything (*:*), and get results and the characters are NOT mangled.
> 
>  * Using the same Perl interface, I try to enter a query using Chinese characters, but I always get zero results.
> 
>  * Using the same Perl interface, I can search for the word "body" (an HTML element I didn't delete), I get expected results.
> 
>  * Using the same Perl interface, I an enter a query using the mangled characters, and I get the sorts of results I expect.
> 
> I believe I have indexed my documents "correctly", but I can't seem to query the index in the expected manner. What might I be doing wrong?
> 
> [1] Solr documentation - https://lucene.apache.org/solr/guide/6_6/language-analysis.html#LanguageAnalysis-TraditionalChinese
> 
> -- 
> Eric Lease Morgan
> Digital Initiatives Librarian, Navari Family Center for Digital Scholarship
> Hesburgh Libraries
> 
> University of Notre Dame
> 250E Hesburgh Library
> Notre Dame, IN 46556
> o: 574-631-8604
> w: cds.library.nd.edu