 |
 |
On Wed, Feb 27, 2013 at 1:42 PM, Andy Kohler <[log in to unmask]> wrote: > I agree with Terry: use a database. Since you're doing multiple queries, > invest the time up front to import your data in a queryable format, with > indexes, instead of repeatedly building comparison files... > > Another, completely unrelated, possible solution depending on your needs: > run the records through solrmarc and do your queries via solr? > > Good luck... let us know what you eventually decide to do. > After trying a few experiments, it appears that my use case (mostly comparing huge record sets with an even bigger record set of records on indexed points) is well suited to a relational model. My primary goal is to help a bunch of libraries migrate to a common catalog so the primary thing people are interested in knowing is what data is local to their catalog. Identifying access points and relevant description in their catalog that are not in the master record involves questions like "Give me a list of records where field X occurs more times in our local catalog than in the master record (or that value is missing from the master record -- thank goodness for LEFT JOIN)" so that arrangements can be made. I'm getting surprising performance and the convenience of being able to do everything from the command line is nice. kyle