 |
 |
On Wed, Oct 25, 2017 at 8:57 AM, Eric Lease Morgan <[log in to unmask]> wrote: > ...My bibliographic data is fraught with inconsistencies. For example, a > publisher’s name may be recorded one way, another way, or a third way. The > same goes for things like publisher place: South Bend; South Bend, IN; > South Bend, Ind. And then there is the ISBD punctuation that is sometimes > applied and sometimes not. All of these inconsistencies make indexing & > faceted browsing more difficult than it needs to be. Effective normalizing is about understanding patterns that represent the same thing and being aware of the patterns associated with specific types of data. For example, in your publisher example, detecting geographic entities and normalizing the states would be easy enough. You'll also see variation in how the publisher names themselves are expressed but also that the vast majority of variations follow a small number of patterns. Don't be afraid to use multifield logic to normalize one or multiple fields. To return to your publisher example, fragments from the publisher name and place may be used to normalize both fields individually and collectively more accurately than attempting to normalize each field in isolation. What is the source of your bib data -- or are their many? You may be able to use info such as byte 18 (descriptive cataloging form) in the Leader or even the cat date to figure out cataloging rules that would have been in play that drive patterns specific to those records. If you have multiple sources of records, the patterns will most likely vary with the source, e.g. there are multiple ways personal names can be expressed, but the number of variations is small. Depending on what you're working with, other clustering tools may be helpful. However, you may get better and more predictable results using a method tuned for the data you have than a much more sophisticated mechanism created for other uses. kyle