Eric Lease Morgan, you are awesome. Thank you so much for your enlightening explanation, with source code, even! 

code4lib is the BEST!

Bess

On Oct 1, 2012, at 7:39 PM, Eric Lease Morgan wrote:

> On Oct 1, 2012, at 7:55 PM, Bess Sadler wrote:
> 
>> For a full-text search system we're prototyping, we are being asked to provide term co-occurrence analysis. I'm not very familiar with this concept, so maybe someone on the list can describe it better, but I believe that what is wanted is to be able to query a text corpus for a given word, and to receive in return a list of words that co-occur with the search term, along with some indication of how often those words co-occur. Something like this IBM Many Eyes demo: http://ibm.co/PT5N59 (but we're not necessarily looking for a visualization, just a way to do the query).
> 
> 
> Interesting, but alas, I do not have a Solr recipe.
> 
> Yes, co-occurance -- along with ngram -- are terms used to denote and infer the distance between different tokens (usually words) in a text. Successful phrase searching assumes some sort of underlying co-occurance algorithm because the indexing process assigns positions to its indexed tokens (words).
> 
> An algorithm creating a list co-occurances is an almost trivial:
> 
>  1. read text
> 
>  2. parse text into individual tokens
>     (words)
> 
>  3. read a token and the next token
> 
>  4. update a list (hash or "associative
>     array") with the pair of tokens
> 
>  5. update a list of the number of times
>     this particular pair of tokens exist
> 
>  6. go to step #3 for each token
> 
>  7. done
> 
> If one wants to read more than bigrams (two-word phrases), then change Step #3 to include the next token and the token after that one -- trigrams.
> 
> Given a particular token, listing the co-occurances of that token and other words is "simply" a matter of searching the list for the token and returning it and its co-occurance. 
> 
> How to do this in Solr? After re-reading the documentation I believe the ord function may be of some use, but I'm not sure:
> 
>  ord(myfield) returns the ordinal of the indexed field value
>  within the indexed list of terms for that field in lucene index
>  order (lexicographically ordered by unicode value), starting at
>  1. In other words, for a given field, all values are ordered
>  lexicographically; this function then returns the offset of a
>  particular value in that ordering. The field must have a maximum
>  of one value per document (not multiValued). 0 is returned for
>  documents without a value in the field.
> 
>    * Example: If there were only three values for a particular field:
>      "apple","banana","pear", then ord("apple")=1, ord("banana")=2,
>      ord("pear")=3
>    * Example Syntax: ord(myIndexedField)
>    * Example SolrQuerySyntax: _val_:"ord(myIndexedField)"
> 
>  http://bit.ly/SiL8eM
> 
> I have a hammer (Perl) and everything to me looks like a nail. Consequently I would use a module I wrote to do this sort of thing -- http://bit.ly/bgmhXM -- specifically the ngram method.
> 
> -- 
> HTH, ELM