Hi Eric,

sounds an interesting project!

You have multiple choices. Hadoop is at least two kind of things: a
distributed file system and a distributed computation engine with its
own API. If you upload files to the file system Hadoop will distribute
them in a safe way. The basic idea of the computation is that once the
files are located, the program is distributed to the files. For
processing you can use the Hadoop API which is based on the MapReduce
paradigm, or you can use some other tools based on Hadoop, I have
experience with Apache Spark. In either way you can read and write
data from the file system or from other sources which has Hadoop or
Spark interface (Cassandra, MongoDB etc.). The basic workflow is that
you submit a .jar file to Hadoop and it distributes the jar to the
three nodes. You can even exploit the multicore nature of these nodes:
Hadoop and Spark are very effective in multithread processing out of
the box and you do not need to put the usual Java multithread handling
parts into your code. The output is also stored the Hadoop File System
by default.

If you plan to use XSEDE I suggest you to check out this project:
https://www.xsede.org/news/science-stories/-/asset_publisher/9JovW1UTN10Q/content/ruby-mendenhall-charts-progress-using-hpc-big-data-to-flag-unidentified-historical-sources-on-african-american-women%E2%80%99s-lives/10165.
Prof. Ruby Mendenhall used XSEDE to do distant reading (for those who
doesn't know this term was coined by Franco Moretti, a Stanford
professor) on HathiTrust and JStore data to reveail traces of black
women lives in the first half of 19th century.

I am happy to share you some more technical tips if needed.

Best,
Péter

Eric Lease Morgan <[log in to unmask]> ezt írta (időpont: 2018. dec. 17., H, 18:53):
>
> What is your experience with Apache Hadoop?
>
> I have very recently been granted root privileges on as many as three virtual machines. Each machine has forty-four cores, and more hard disk space & RAM than I really know how to exploit. I got access to these machines to work on a project I call The Distant Reader, and The Distant Reader implements a lot of map/reduce computing.†
>
> Can use Apache Hadoop to accept jobs on one machine, send it to any of the other two machines, and then save the results in some sort of common/shared file system?
>
> † In reality, The Distant Reader is ultimately intended to be an XSEDE science gateway --> https://www.xsede.org. The code for the Reader is available on GitHub --> https://github.com/ericleasemorgan/reader
>
> --
> Eric Morgan
> University of Notre Dame



-- 
Péter Király
software developer
GWDG, Göttingen - Europeana - eXtensible Catalog - The Code4Lib Journal
http://linkedin.com/in/peterkiraly