 |
 |
On Fri, May 9, 2008 at 11:14 AM, Bess Sadler <[log in to unmask]> wrote: > > Casey, you say you're getting indexing times of 1000 records / > second? That's amazing! I really have to take a closer look at > MarcThing. Could pymarc really be that much faster than marc4j? Or > are we comparing apples to oranges since we haven't normalized for > the kinds of mapping we're doing and the hardware it's running on? > Well, you can't take a closer look at it yet, since I haven't gotten off my lazy butt and released it. We're still using an older version of the project in production on LT. I'm going to cut us over to the latest version this weekend. At this point, being able to say we eat our own dogfood is the only barrier to release. The last indexer I wrote (the one used by fac-back-opac) used marc4j and was around 100-150 a second. Some of the boost was due to better designed code on my end, but I can't take too much credit. Pymarc is much, much faster. I never bothered to figure out why. (That wasn't why I switched, though -- there are some problems with parsing ANSEL with marc4j (*) which I decided I'd rather be mauled by bears than try and fix -- the performance boost was just a pleasant surprise). Of course one could use pymarc from java with Jython. Undoubtedly we're comparing apples to oranges here. 1000/sec. is about what I can get on my Macbook Pro on some random MARC records I have lying around, with plenty of hand-waving involved. MARCThing does do a fair amount of munging for expanding codes, guessing physical format and what-have-you (but nothing with dates, which is sorely needed), but I think it would be a bad idea to read too much into some anecdotal numbers. --Casey (*) in marc4j's defense, actually due to a bug in Horizon ILS.