 |
 |
Thanks, everyone! It is always nice to hear what others do, even if it doesn't end up being what works for us. On Thu, Jan 28, 2016 at 4:06 AM, Owen Stephens <[log in to unmask]> wrote: > To share the practice from a project I work on - the Jisc Historical Texts > platform[1] which provides searching across digitised texts from the 16th > to 19th centuries. In this case we had the option to build the search > application from scratch, rather than using a product such as ContentDM > etc. I should say that all the technical work was done by K-Int [2] and > Gooii [3], I was there to advise on metadata and user requirements, and so > the following is based on my understanding of how the system works, and any > errors are down to me :) > > There are currently three major collections within the Historical Texts > platform, with different data sources behind each one. In general the data > we have for each collection consists of MARC metadata records, full text in > XML documents (either from transcription or from OCR processes) and image > files of the pages. > > The platform is build using the ElasticSearch [4] (ES) indexing software > (as with Solr this is built on top of Lucene). > > We structure the data we index in ES in two layers - the ‘publication’ > record, which is essentially where all the MARC metadata lives (although > not as MARC - we transform this to an internal scheme), and the ‘page’ > records - one record per page in the item. The text content lives in the > page record, along with links to the image files for the page. The ‘page’ > records are all what ES calls ‘child’ records of the relevant publication > record. We make this relationship through shared IDs in the MARC records > and the XML fulltext documents. > > We create a whole range of indexes from this data. Obviously field > specific searchs like title or author only search the relevant metadata > fields. But we also have a (default) ’search all’ option which searches > through all the metadata and fulltext. If the user wants to search the text > only, they check an option and we limit the search to only text from > records of the ‘page’ type. > > The results the user gets initially are always the publication level > records - so essentially your results list is a list of books. For each > result you can view ‘matches in text’ which shows snippets of where your > search term appears in the fulltext. You can then either click to view the > whole book, or click the relevant page from the list of snippets. When you > view the book, the software retrieves all the ‘page’ records for the book, > and from the page records can retrieve the image files. When the user goes > to the book viewer, we also carry over the search terms from their search, > so they can see the same text snippets of where the terms appear alongside > the book viewer - so the user can navigate to the pages which contain the > search terms easily. > > For more on the ES indexing side of this, Rob Tice from Knowledge > Integration did a talk about the use of ES in this context at the London > Elasticsearch usergroup [5]. Unfortunately the interface itself requires a > login, but if you want to get a feel for how this all works in the UI, > there is also a screencast which gives an overview of the UI available [6]. > > Best wishes, > > Owen > > 1. https://historicaltexts.jisc.ac.uk > 2. http://www.k-int.com > 3. http://www.gooii.com > 4. https://www.elastic.co > 5. > http://www.k-int.com/Rob-Tice-Elastic-London-complex-modelling-of-rich-text-data-in-Elasticsearch > 6. http://historicaltexts.jisc.ac.uk/support > > Owen Stephens > Owen Stephens Consulting > Web: http://www.ostephens.com > Email: [log in to unmask] > Telephone: 0121 288 6936 > > > On 27 Jan 2016, at 00:30, Laura Buchholz <[log in to unmask]> > wrote: > > > > Hi all, > > > > I'm trying to understand how digital library systems work when there is a > > need to search both metadata and item text content (plain text/full > text), > > and when the item is made up of more than one file (so, think a digitized > > multi-page yearbook or newspaper). I'm not looking for answers to a > > specific problem, really, just looking to know what is the current state > of > > community practice. > > > > In our current system (ContentDM), the "full text" of something lives in > > the metadata record, so it is indexed and searched along with the > metadata, > > and essentially treated as if it were metadata. (Correct?). This causes > > problems in advanced searching and muddies the relationship between what > is > > typically a descriptive metadata record and the file that is associated > > with the record. It doesn't seem like a great model for the average > digital > > library. True? I know the answer is "it depends", but humor me... :) > > > > If it isn't great, and there are better models, what are they? I was > taught > > METS in school, and based on that, I'd approach the metadata in a METS or > > METS-like fashion. But I'm unclear on the steps from having a bunch of > METS > > records that include descriptive metadata and pointers to text files of > the > > OCR (we don't, but if we did...) to indexing and providing results to > > users. I think another way of phrasing this question might be: how is the > > full text of a compound object (in the sense of a digitized yearbook or > > similar) typically indexed? > > > > The user requirements for this situation are essentially: > > 1. User can search for something and get a list of results. If something > > (let's say a pamphlet) appears in results based on a hit in full text, > the > > user selects the pamphlet which opens to the file (or page of the > pamphlet) > > that contains the text that was matched. This is pretty normal and does > > work in our current system. > > 2. In an advanced search, a user might search for a name in the "author" > > field and a phrase in the "full text" field, and say they want both > > conditions to be fulfilled. In our current system, this won't provide > > results when it should, because the full text content is in one record > and > > the author's name is in another record, so the AND condition can't be > met. > > 3. Librarians can link description metadata records (DC in our case) to > > particular files, sometimes one to one, sometimes many to one, sometimes > > one to many. > > > > If this is too unclear, let me know... > > Thanks! > > > > -- > > Laura Buchholz > > Digital Projects Librarian > > Reed College Library > > 503-517-7629 > > [log in to unmask] > -- Laura Buchholz Digital Projects Librarian Reed College Library 503-517-7629 [log in to unmask]