On 11/28/06, Erik Hatcher <[log in to unmask]> wrote:

> Is there a standard for specifying how textual analysis works as
> well, so that tokenization can be standardized across these XQuery
> engines as well?

Not that I know.  What I've seen so far is that tokenization is
implementation specific.  Perhaps this is something that is
configurable so that implementations can be set up and then queried
consistently.  Any indexing engine worth its salt should be
configurable I'd think.  There is nothing I'm aware of in the fulltext
work though that defines how things are indexed.

> That's an easy bet... of course Lucene will be part of it.  It's
> already implemented as extensions to XQuery engines (Nux, I know of,
> and surely others).

As you can tell, I'm not really a gambler :-)

Our native XML database vendor has committed to the fulltext spec
(once it becomes a spec) and since they are using Lucene already I'd
say I don't have anything to worry about.

Interestingly, as a side note, a quick search turned up an eXist
presentation from Prague06 saying that eXist's text analysis classes
would be replaced by a "modular analyzer provided by Apache's Lucene."
 Neat.

All this talk is just me looking forward (with optimism).  It is
possible to use fulltext with XQuery now either through an
intermediary layer like we currently have (Lucene search is done and
the results passed to XQuery and our native XML database for retrieval
and munging) or by creating fulltext extensions (like eXist db and our
native XML database vendor have done).

Personally, I wish we had taken the extension route, but it was just
quicker for me to do something in Java and have the search and XQ
servlets chain rather than adding the extra extension layer through
our XQuery processor.  Quicker isn't always better/cleaner/nicer
though...

Kevin