I've had sight of this and generated a dictionary from the LoC bib data. It's very fast and the suggestions are excellent, including multi-word corrections.

rob

Here's Martin's mail with the details - but I encourage you to join the group
----
But I'm sure you'd like to get your hands on it, so I packaged up a convenient way to quickly make a dictionary from your own data files, and run your own queries through it.

Grab this jar: http://groups.google.com/group/spelt/web/spelt.jar

Everything is included (Spelt and Lucene). The only requirement is that you have JDK 1.5 (5.0) -- it won't work with JDK 1.4.

The utility includes a fast-but-dumb text ripper that does a deep directory scan for textual files, and pulls out all the words. It should be able to handle XML, HTML, and plain text files (provided they're in UTF-8 encoding.) You can build a dictionary this way:

   java -jar spelt.jar -build <your-src-dir> speltDictDir

If you run it on a big data set, I'd suggest giving it more RAM, like this:

   java -Xmx750m -jar spelt.jar -build <your-src-dir> speltDictDir

You can run a set of test queries (e.g, http://groups.google.com/group/spelt/web/test.list) like this:

   java -jar spelt.jar -test speltDictDir test.list

Finally, if you are curious about how this compares with the exisitng code in Lucene, you can add the -old flag just before "-build" or "-test". Warning: the build process is about 35 times slower on my machine, so I'd suggest doing this on a small data set. 


-----Original Message-----
From: Code for Libraries on behalf of Jonathan Rochkind
Sent: Tue 03/04/2007 7:01 PM
To: [log in to unmask]
Subject: Re: [CODE4LIB] pspell aspell: make your own word lists/dictionaries
 
I haven't had time to look at it yet, but someone at Code4Lib conference
proposed a more sophisticated approach to spell checking that sounded
really interesting to me, and said he was going to share the code. I
hope to have time to investigate at some point.

Let's see if I can find it on the conference page.... yeah, it was
Martin Haye. You can watch his presentation here:
http://video.google.com/videoplay?docid=4028600349627496246&hl=en

Looks like he's *martin*.*haye*[at]gmail.com.  During the lightning
talk, he said he didn't want to distribute the code seperately but
wanted to include it in Lucene if possible---but later in the
conference, he said he had been convinced by the interest in it to
distriburte the code as it's own standalone thing, and planned to do
that presently.

If anyone does or has explored using martin's code, please let us know
about your experience.

Jonathan

Kevin Kierans wrote:
> Has anyone created their own "dictionaries"
> for aspell?  We've created blank delimited
> lists of words from our opac.  One for title,
> one for subjects, and one for authors.  (We're thinking
> of a series one as well)
>
> We would like to use
> one of these word lists to offer suggestions
> depending on which search the patron is making.
> We're assuming we can make better suggestions
> if the words come from our actual opac.
>
> We've got it working with the dictionary that
> comes with aspell, but having problems (we can't do it!)
> substituting our own  "dictionaries."
>
> Does anyone have any experience/knowledge/hints/pointers
> they can share with us?
>
> We are using linux, php 5,  aspell 0.50.5, and
> php -> pspell functions.
>
> Thanks,
> Kevin
> TNRD Library System, Kamloops, British Columbia, Canada
>
>

--
Jonathan Rochkind
Sr. Programmer/Analyst
The Sheridan Libraries
Johns Hopkins University
410.516.8886
rochkind (at) jhu.edu
 
The very latest from Talis
read the latest news at www.talis.com/news 
listen to our podcasts www.talis.com/podcasts 
see us at these events www.talis.com/events 
join the discussion here www.talis.com/forums 
join our developer community www.talis.com/tdn 
and read our blogs www.talis.com/blogs

 
Any views or personal opinions expressed within this email may not be those of Talis Information Ltd. The content of this email message and any files that may be attached are confidential, and for the usage of the intended recipient only. If you are not the intended recipient, then please return this message to the sender and delete it. Any use of this e-mail by an unauthorised recipient is prohibited.
 
 
Talis Information Ltd is a member of the Talis Group of companies and is registered in England No 3638278 with its registered office at Knights Court, Solihull Parkway, Birmingham Business Park, B37 7YB.