See also http://wiki.tei-c.org/index.php/Heuristics , which discusses 
this problem more broadly conceived.  I've just added a link to the 
archives of this very discussion.  --Kevin

On 6/18/15 12:52 PM, Matt Sherman wrote:
> The hope is to take these bibliographies put it into more of a web
> searchable/sortable format for researchers to make use out of them.  My
> colleague was taking some inspiration from the Marlowe Bibliography (
> https://marlowebibliography.org/), though we are hoping to possibly get a
> bit more robust with the bibliography we are working on.  The important
> first step it to be able to parse the existing OCRed bibliography scans we
> have into a database, possibly a custom XML format but a database will
> probably be easier to append and expand down the road.
>
> On Thu, Jun 18, 2015 at 1:11 PM, Kyle Banerjee <[log in to unmask]>
> wrote:
>
>> How you want to preprocess and structure the data depends on what you hope
>> to achieve. Can you say more about what you want the end product to look
>> like?
>>
>> kyle
>>
>> On Thu, Jun 18, 2015 at 10:08 AM, Matt Sherman <[log in to unmask]>
>> wrote:
>>
>>> That is a pretty good summation of it yes.  I appreciate the suggestions,
>>> this is a bit of a new realm for me and while I know what I want it to do
>>> and the structure I want to put it in, the conversion process has been
>>> eluding me so thanks for giving me some tools to look into.
>>>
>>> On Thu, Jun 18, 2015 at 1:04 PM, Eric Lease Morgan <[log in to unmask]>
>> wrote:
>>>
>>>> On Jun 18, 2015, at 12:02 PM, Matt Sherman <[log in to unmask]>
>>>> wrote:
>>>>
>>>>> I am working with colleague on a side project which involves some
>>> scanned
>>>>> bibliographies and making them more web
>>> searchable/sortable/browse-able.
>>>>> While I am quite familiar with the metadata and organization aspects
>> we
>>>>> need, but I am at a bit of a loss on how to automate the process of
>>>> putting
>>>>> the bibliography in a more structured format so that we can avoid
>> going
>>>>> through hundreds of pages by hand.  I am pretty sure regular
>>> expressions
>>>>> are needed, but I have not had an instance where I need to automate
>>>>> extracting data from one file type (PDF OCR or text extracted to Word
>>>> doc)
>>>>> and place it into another (either a database or an XML file) with
>> some
>>>>> enrichment.  I would appreciate any suggestions for approaches or
>> tools
>>>> to
>>>>> look into.  Thanks for any help/thoughts people can give.
>>>>
>>>>
>>>> If I understand your question correctly, then you have two problems to
>>>> address: 1) converting PDF, Word, etc. files into plain text, and 2)
>>>> marking up the result (which is a bibliography) into structure data.
>>>> Correct?
>>>>
>>>> If so, then if your PDF documents have already been OCRed, or if you
>> have
>>>> other files, then you can probably feed them to TIKA to quickly and
>>> easily
>>>> extract the underlying plain text. [1] I wrote a brain-dead shell
>> script
>>> to
>>>> run TIKA in server mode and then convert Word (.docx) files. [2]
>>>>
>>>> When it comes to marking up the result into structured data, well, good
>>>> luck. I think such an application is something Library Land sought for
>> a
>>>> long time. “Can you say Holy Grail?"
>>>>
>>>> [1] Tika - https://tika.apache.org
>>>> [2] brain-dead script -
>>>> https://gist.github.com/ericleasemorgan/c4e34ffad96c0221f1ff
>>>>
>>>> —
>>>> Eric
>>>>
>>>
>>