 |
 |
On Jul 18, 2007, at 10:04 PM, Eric Hellman wrote: > Anyway, almost all parsers rely on a set of heuristics. I have not > seen any parsers that do a good job of managing their heuristics in a > scaleable way. A successful open-source attack on this problem would > have the following characteristics: > 1. able to efficiently handle and manage large numbers of parsing and > scoring heuristics > 2. easy for contributors to add parsing and scoring heuristics > 3. able to use contextual information (is the citation from a physics > article or from a history monograph?) in application and scoring of > heuristics One of the more problematic things is that we don't always get the contextual information as to where citations occurred -- in fact, it's quite rare to get that. Also, even in (many) scholarly journals, editorial consistency is almost unbelievably poor -- lots of times, the rules just aren't followed. Punctuation gets missed, journal names (especially abbreviations!) are misspelled... and so on. Rule-based and heuristic systems are always going to have problems in those cases. In a lot of ways, I think the problem is fundamentally similar to identifying parts of speech in natural language (which has lots of the same ambiguities) -- and the same techniques that succeed there will probably yield the most robust results for citation parsing. -n