On Mar 20, 2020, at 11:40 AM, Eric Lease Morgan <[log in to unmask]> wrote:

> https://carrels.distantreader.org/library/covid-19/
> https://carrels.distantreader.org/covid.cgi


Wanted: experts in the creation and deployment of Solr indexes and experts in relational database technology

The sponsors of the Distant Reader, the good folks at XSEDE, are very strongly encouraging me to submit a fast-track proposal to: 1) increase the size of the Distant Reader's underlying high performance computing system, and 2) apply the technology to the growing Coronavirus literature. To that end, I have outlined a set of tasks/functions, attached.

But the work requires deeper knowledge of Solr and relational databases than what I possess. For example, we will be wanting to create a Solr instance to index 10s of thousands of scholarly journal articles, if not just as many sets of non-scholarly materials. Similarly, we will be wanting to analyze the literature in terms of its ngrams, parts-of-speech, named entities, and grammars. These things can/will be saved in both tabular forms as well as a relational database. Much of this work has already been done, but it needs to scale up a few notches. We need your help. 

If you would like to apply your skills in a high performance computing environment, then you might want to participate in a Zoom meeting scheduled for Friday at 9 AM (EDT). Drop me a line to ask questions and/or "raise your hand" if you would like to participate in the meeting. 

Be safe.

[1] XSEDE - https://www.xsede.org

--
Eric Lease Morgan
University of Notre Dame






The Distant Reader Meets Covid-19

Below is a list of tasks/functions which we can implement as a part of a proposal. Many of these tasks/functions may be implemented concurrently, and they are barely prioritized:

 1. Dedicate a single Reader node to serving a full text index of
    virus-related materials, and these materials will initially focus
    on the CORD-19 journal literature dataset. This machine would do
    very nicely at only two or four cores.

 2. Dedicate a few nodes to harvesting and indexing the data for
    Item #1. Indexing is a bit computing heavy.

 3. If Items #1 and #2 are successful, then index additional, but
    more difficult to acquire, journal literature, like content from
    JSTOR or Zotero libraries

 4. Identify the likes of Team JAMS (the good students who won the
    PEARC hack-a-thon), and have these people use the Reader's output
    (ngrams, parts-of-speech, grammars, named-entities, etc) as the
    input for things like discovering relationships between drugs,
    "correlations of language" between articles, or visualizations of
    the underlying data such timelines, geographic maps, or network
    diagrams.

 5. Modify the Reader's code to use a biomedical language model
    instead of the existing English language model.

 6. Modify the Reader's code so the feature-extraction tasks are
    more distinct from the report generation tasks, thus we can
    divide and conquer when it comes to report generation.

 7. Work on a subset of the CORD dataset, and get the subset
    working

 8. If Items #5, #6, and #7 are successful, then increase the
    carrel's content to include all the CORD content.

 9. If Item #8 is successful, then include content from Item #3

10. Implement a better, more interactive topic modeling
    interface. Just as everybody likes to search, topic modeling is
    very popular in text studies.

11. Integrate the full text indexing and the topic modeling
    interface into the Reader's study carrel thus creating a coherent
    whole. For example: search index, create subset, and topic model
    it. For example: peruse study carrel, identify thing of interest,
    link it to full text index or topic model, and return thing of
    interest in the context of the original article. Etc.


What might be needed to do this work? Some of it might include:

 1. A re-allocation of existing cores, thus some systems
    administration

 2. A re-examination of the shared file system because my
    antidotal observations see a lot of time is spent on disk I/O

 3. Hacker(s) who can read delimited files over the 'Net (or a
    relational database file), parse it, ask questions of it, and
    visualize the results.

 4. Content experts who can evaluate the output of everything above.

 5. Time.


Here is a list of nice-to-have items -- people:

 1. Someone who really knows Solr -- the full text indexer of
    choice

 2. Someone who really knows relational databases -- because a
    whole lot of the data is ultimately stored in one

 3. Someone who can write interactive Web-pages to... interact
    with the underlying data in real time.

 4. People to create additional study carrels of related content,
    and then we can meld the resulting carrels togehter.


The above is only a set of suggestions. I hope they give you ideas, and I hope you are available to chat on Friday at 9 o'clock. 

In any event, additional suggestions and reactions are welcome.

--
Eric Morgan and Team Reader
March 23, 2020