 |
 |
On Fri, May 09, 2008 at 11:58:03AM -0700, Casey Durfee wrote: > On Fri, May 9, 2008 at 11:14 AM, Bess Sadler <[log in to unmask]> wrote: > > > > > Casey, you say you're getting indexing times of 1000 records / > > second? That's amazing! I really have to take a closer look at > > MarcThing. Could pymarc really be that much faster than marc4j? Or > > are we comparing apples to oranges since we haven't normalized for > > the kinds of mapping we're doing and the hardware it's running on? > > > > Well, you can't take a closer look at it yet, since I haven't gotten off my > lazy butt and released it. We're still using an older version of the > project in production on LT. I'm going to cut us over to the latest version > this weekend. At this point, being able to say we eat our own dogfood is > the only barrier to release. Looking forward to the release. I'd be interested to see how it compares to the pymarc-indexer branch in FBO/Helios [1]. > The last indexer I wrote (the one used by fac-back-opac) used marc4j and was > around 100-150 a second. Some of the boost was due to better designed code > on my end, but I can't take too much credit. Pymarc is much, much faster. > I never bothered to figure out why. (That wasn't why I switched, though -- > there are some problems with parsing ANSEL with marc4j (*) which I decided > I'd rather be mauled by bears than try and fix -- the performance boost was > just a pleasant surprise). Of course one could use pymarc from java with > Jython. On the small set of documents I'm now indexing (3327) I get 141 rec/sec. This is on my test server, an AMD64 whose processor speed I can't recall. That rate includes pymarc processing (~65%) and the loading of the CSV file into SOLR (~35%). Surely there's some room there for optimization, but it's fast enough for my current purposes. Also, I'm in the camp that would be happy with a ~10,000 record test set. There will always be some edge cases that we'll only solve as they're encountered. I need rapid iteration! Gabriel [1] http://fruct.us/trac/fbo/browser/branches/pymarc-indexer/indexer