Thanks, Erik, there is no specific reason for their removal, I think this was just that the StopFilterFactory is preconfigured in the analyzer chain for fieldType=text. We will do some performance testing with this filter removed.
BTW, a useful tool in deciding appropriate stopwords is the schema browser, which can be found on the /solr/admin page. Here you can see term frequencies for each of the fields sorted from highest frequency to help weed out the terms of little querying value.
Eric
> Date: Thu, 12 Nov 2009 09:06:46 -0500
> From: [log in to unmask]
> Subject: Re: [CODE4LIB] solr | StopFilterFactory - stopwords.txt
> To: [log in to unmask]
>
> I often recommend against stop word removal altogether. Is there any
> reason you need to remove them?
>
> The primary reason stop words get removed is to increase performance
> of queries with very common terms. If you are encountering that,
> using Solr's CommonGramsFilter(Factory) is a good solution to keep
> your stop words and alleviate the performance degradation potential.
> The HathiTrust folks have had success with the common grams capability.
>
> Erik
>
>
> On Nov 11, 2009, at 3:41 PM, Eric James wrote:
>
> > Has anyone already given some thought into refining the solr
> > stopwords.txt for library collections, particularly finding aids?
> > The words included in the out of the box stopwords.txt are of very
> > questionable unimportance:
> >
> > <an and are as at be but by for if in into is it not of on or s such
> > t that the their then there these they this to was will with>
> >
> >
> >
> > We were indexing a field id with "no." as one of its tokens (for
> > number), but wanted a query with "no" (where the person did not add
> > the period) to find the doc, but in actuality the "no" would get
> > stripped by the StopFilterFactory. And thus we stumbled upon this
> > list, and was a bit suprised by some of the inclusions (ex:"will"),
> > and exclusions( ex:"a").
> >
> >
> >
> > Thanks,
> >
> > Eric James
> >
> > Yale University Libraries
> >
|
