On Jul 18, 2007, at 10:04 PM, Eric Hellman wrote:

> Anyway, almost all parsers rely on a set of heuristics. I have not
> seen any parsers that do a good job of managing their heuristics in a
> scaleable way. A successful open-source attack on this problem would
> have the following characteristics:
> 1. able to efficiently handle and manage large numbers of parsing and
> scoring heuristics
> 2. easy for contributors to add parsing and scoring heuristics
> 3. able to use contextual information (is the citation from a physics
> article or from a history monograph?) in application  and scoring of
> heuristics

One of the more problematic things is that we don't always get the
contextual information as to where citations occurred -- in fact,
it's quite rare to get that.

Also, even in (many) scholarly journals, editorial consistency is
almost unbelievably poor -- lots of times, the rules just aren't
followed. Punctuation gets missed, journal names (especially
abbreviations!) are misspelled... and so on. Rule-based and heuristic
systems are always going to have problems in those cases.

In a lot of ways, I think the problem is fundamentally similar to
identifying parts of speech in natural language (which has lots of
the same ambiguities) -- and the same techniques that succeed there
will probably yield the most robust results for citation parsing.

-n