 |
 |
Casey, we have had great successes with XSL for MARCXML to SOLR, so I can't agree to everything you are saying. However I anxiously await your presentation on your successes with SOLR so you can persuade me to the dark side :) Casey Durfee wrote: > > I think there are many good reasons why XSLT is absolutely the wrong tool for the job of indexing MARC records for Solr. > > 1) Performance/Speed: In my experience even just transforming from MARCXML to MODS takes a second or two (using the LoC stylesheet), due to the stylesheet's complexity and inefficiency of doing heavy-duty string manipulation in XSL. That means you're looking at an indexing speed of around 1 record/second. If you've got 1,000,000 bib records, it'll take a couple of weeks just to index your data. For comparison, the indexer of our commercial OPAC does about 50 records per second (~6 hours for a million records) and the one I've written in Jython (by no means the fastest language out there) that doesn't use XSL can do about 150 records a second (about 2 hours for 1 million records). > I can transform 500,000 records from marcxml to solrxml in about 4 hours. Then about 2 hours for importing to SOLR. Considering time is NOT truly a factor, I think at this point it is totally based on developer preference (assuming your XSLT process is not 2 weeks long). Once you have your records in SOLR, you are all set. You only need to re-run your transformation on a periodic basis to catch records that change. In our instance that might only be 5 - 10 records per day. > > 2) Reusability: What if you want to change how a field is indexed? You would have to edit the XSLT directly (or have the XSL stylesheet automatically generated based on settings stored elsewhere). > > a) Users of the indexer shouldn't have to actually mess with programming logic to change how it indexes. You shouldn't have to know a thing about programming to change the setup of an index. > > b) It should be easy for an external application to know how your indexes have been built. This would be very difficult with an XSL stylesheet. Burying configuration inside of programming logic is a bad idea. > > c) The Solr schema should be automatically generated from your index setup so all your index configuration is in one place. I guess you could write *another* XSL stylesheet that would transform your indexing stylesheet into the Solr schema file, but that seems ridiculous. > > d) Automatic code generation is evil. Blanchard's law: "Systems that require code generation lack sufficient power to tackle the problem at hand." If you find yourself considering automatic code generation, you should instead be considering a more dynamic programming language. > I agree with your argument of abstracting your programming from your data so that a non-tech-savvy librarian could modify the solr settings. But if you modify the solr settings, you need to (at this point) reimport all of your data which mean that you either have to change your XSLT or your transformation application. I personally feel that a less-tech savvy individual can pickup XSLT easier than coding java. Maybe I am understanding you incorrectly though. > > 3) Ease of programming. > > a) Heavy-duty string manipulation is a pain in pure XSLT. To index MARC records have to do normalization on dates and names and you probably want to do some translation between MARC codes and their meanings (for the audience & language codes, for instance). Is it doable? Yes, especially if you use XSL extension functions. But if you're going to have huge chunks of your logic buried in extension functions, why not go whole hog and do it all outside of XSLT, instead of having half your programming logic in an extension function and half in the XSLT itself? > I can see your argument for this, however I like to abstract my layers of applications as mentioned above. So in this aspect, I have a script the runs the XSLT. Inside the script is also some logic that the XSLT refers back to for the manipulation and massaging of the data. I can keep all XML related transformation logic in my XSL and all of my coding logic in my script. Again, I think it boils down to preference. > > b) Using XSLT makes object-oriented programming with your data harder. That's a bold statement. > Your indexer should be able to give you a nice object representation of a record (so you can use that object representation within other code). If you go the XSLT route, you'd have to parse the MARC record, transform it to your Solr record XML format, then parse that XML and map the XML to an object. If you avoid XSLT, you just parse the MARC record and transform it to an object programmatically (with the object having a method to print itself out as a Solr XML record). > > Honestly, all this talk of using XSLT for indexing MARC records reminds me of that guy who rode across the United States on a riding lawnmower. I am looking forward to there being a standard, well-tested MARC record indexer for Solr (and would be excited to contribute to such a project), but I don't think that XSL is the right tool to use. > I can agree with your OO style of design in which you have one Record object that is responsible for all of the work (converting to solr, and back again) but again, this all seems to be based on preference. I have an import script that is completely independent of our SOLR libraries. I have a main SOLR class that is responsible for interacting with SOLR and as well creating Record objects (using XSLT of course). Also, I am sure there are plenty of folks attending the SOLR preconference who are not experienced software developers and may have an easier time developing some XSLT stylesheets -- and an even easier time if we come up with a standard xslt doc for marcxml -> solr -- than learning how to do what you are describing so they can create a nice search engine for their catalog. I feel that your arguments for not using XSLT are based on preference and do not lend toward a "better" design at this point. But I love to be proved wrong ... Im currently finishing up my masters in Computer Science/Software Engineering ... so I love these kinds of debates since they are all thats on my head at the moment. Andrew