 |
 |
> Sounds like you have some experience of this, Kyle! > Do you have a list of "the screwball stuff"? Even an offhand one would > be interesting... I don't have the list with me, but just to rattle a few things off, some extra short records rank high because so much of a search term matches the whole document. Some records contain fields that have been repeated many times which artificially boosts them. You'll see nonstandard use of fields as well as foreign character sets. There are a number of ways URLs are displayed. Deriving format can be problematic because encoded mat type and what you're providing access to are different. Some records contain lots of added entries, while many important ones are fairly minimalist. There are conversion on the fly, purchased record sets, automatically generated ones, full level, ones that automatically have some subject heading added that contains a common search term. There are a zillion other things, but you get the idea. > What sorts of normalizations do you do? I'm starting to look for > standard measures of data quality or data validation/normalization > routines for MARC. Index terms only once. This helps deal with repeated terms in repetitive subject headings and added entries. Look at presence of fields to assign additional material types (particularly useful for electronic resources since these typically have paper records -- but don't be fooled by links to TOC and stuff that's not full text). Give special handling for serials. Keywords need to be weighted differently depending on where they're from (e.g. title worth more than subject). We also assigned a rough "record quality" score based presence/absence of fields so that longer more complete records don't become less important simply because a search term matches less of them than a short record. Give a bit more weight to a true full text retrieval. Number of libraries holding the item is considered. When indexing, 650|a is more important than |x, |y, or |z. Don't treat 650|z the same way as 651|a. Recognize that 650|v is a form, that some common 650|x fields should be treated this way (and neither should just be a regular index term). The only thing we didn't use that a lot of places put a lot of weight on is date -- this is good for retrieving popular fiction, but you have to be really careful with it in academic collections because it can hide classic stuff that's been around a long time. I can't remember everything off the top of my head, but there's a lot and it makes a big difference. kyle