Hi Kyle - 

We did a series of webinars on this last year:  http://duraspace.org/taxonomy/term/188

Declan

-----Original Message-----
From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of Kyle Banerjee
Sent: Wednesday, July 23, 2014 2:29 PM
To: [log in to unmask]
Subject: [CODE4LIB] Publishing large datasets

We've been facing increasing requests to help researchers publish datasets.
There are many dimensions to this problem, but one of them is applying appropriate metadata and mounting them so they can be explored with a regular web browser or downloaded by expert users using specialized tools.

Datasets often are large. One that we used for a pilot project contained well over 10,000 objects with a total size of about 1 TB. We've been asked to help with much larger and more complex datasets.

The pilot was successful but our current process is neither scalable nor sustainable. We have some ideas on how to proceed, but we're mostly making things up. Are there methods/tools/etc you've found helpful? Also, where should we look for ideas? Thanks,

kyle