On Nov 23, 2007, at 7:50 AM, Michael Lackhoff wrote:
>> You can also bring data into Solr using the CSV importer.  I highly
>> recommend folks take a good look at this route.  It's clean, easy,
>> fast:
>> <http://wiki.apache.org/solr/UpdateCSV>
>
> That sounds like what I need. Only problem I see: what about
> escapes? I
> don't know my data good enough to be sure that any possible delimiter
> will never occur within the data. Most exotic characters will probably
> be errors but I still don't want SOLR to choke on it.
> Can I use escapes for separator and/or encapsulator? If so is it \" or
> "" (backslash or doubling)? I found nothing in the docs about it.

I prefer tab-delimited files myself.  Tabs are those worthless
characters that actually hold great value as a separator, only as a
field separator.  Ever.

At the bottom of that wiki page you'll see how to do it with tab
delimited files.   But as you're creating that data, ensure that your
field data is void of tabs except as a separator.  Then you're in
business.  That beats having to worry about quotes and commas and
escaping.

>> For 700,000 records, one first nice step to try is to convert that
>> data into CSV and feed it into Solr.  Create a CSV file on the file
>> system with all those records and use the CSV importer.  I think
>> you'll find that the absolute fastest way to bring data in.   But
>
> It even looks like the direct way (almost) without HTTP since the file
> is read directly from the file system and doesn't have to be squeezed
> through a socket connection.

Right - you can feed the CSV data to it as a path to a local (to the
Solr server) file, or stream the data in via HTTP.

But again, don't concern yourself too much at this stage with the
overhead of HTTP.  Most have found it not to be the bottleneck in big
indexing, especially since the indexer code is running on the Solr
server itself or on a local network.  But the CSV route takes that
out of the picture and provides a very clean and flexible conduit
into Solr.

I call it "column separated values", now that I've finally understood
the real reason for the existence of tabs.

        Erik