@John - Thanks, I'd be interested to learn more about the supportable
pattern you mentioned if there are any readings you'd recommend.

@Joe - Cheers, Andreas Rauber's presentation sounds particularly relevant.
Do you have a link?

@Colin - Thanks for the feedback, I do plan to take a closer look at JIRA.

Dave

On Fri, Mar 13, 2015 at 11:49 PM, Joe Hourcle <[log in to unmask]
> wrote:

>
>
> On Wed, 11 Mar 2015, davesgonechina wrote:
>
>  Hi John,
>>
>> Good question - we're taking in XLS, CSV, JSON, XML, and on a bad day PDF
>> of varying file sizes, each requiring different transformation and audit
>> strategies, on both regular and irregular schedules. New batches often
>> feature schema changes requiring modification to ingest procedures, which
>> we're trying to automate as much as possible but obviously require a human
>> chaperone.
>>
>> Mediawiki is our default choice at the moment, but then I would still be
>> looking for a good workflow management model for the structure of the
>> wiki,
>> especially since in my experience wikis are often a graveyard for the best
>> intentions.
>>
>
>
> A few places that you might try asking this question again, to see if you
> can find a solution that better answers your question:
>
>
> The American Society for Information Science & Technology's Research Data
> Access & Preservation group.  It has a lot of librarians & archivists in
> it, as well as people from various research disiplines:
>
>         http://mail.asis.org/mailman/listinfo/rdap
>         http://www.asis.org/rdap/
>
> ...
>
> The Research Data Alliance has a number of groups that might be relevant.
> Here are a few that I suspect are the best fit:
>
>         Libraries for Research Data IG
>         https://rd-alliance.org/groups/libraries-research-data.html
>
>         Reproducibility IG
>         https://rd-alliance.org/groups/reproducibility-ig.html
>
>         Research Data Provenance IG
>         https://rd-alliance.org/groups/research-data-provenance.html
>
>         Data Citation WG
>         (as this fits into their 'dynamic data' problem)
>         https://rd-alliance.org/groups/data-citation-wg.html
>
> ('IG' is 'Interest Group', which are long-lived.  'WG' is 'Working Group'
> which are formed to solve a specific problem and then disband)
>
> The group 'Publishing Data Workflows' might seem to be appropriate but
> it's actually 'Workflows for Publishing Data' not 'Publishing of Data
> Workflows' (which falls under 'Data Provenance' and 'Data Citation')
>
> There was a presentation at the meeting earlier this week by Andreas
> Rauber in the Data Citation group on workflows using git or SQL databases
> to be able to track appending or modification for CSV and similar ASCII
> files.
>
> ...
>
> Also, I would consider this to be on-topic for Stack Exchange's "Open
> Data" site  (and I'm one of the moderators for the site):
>
>         http://opendata.stackexchange.com/
>
> -Joe
>
>
>
>
>
>
>  On Tue, Mar 10, 2015 at 8:10 PM, Scancella, John <[log in to unmask]> wrote:
>>
>>  Dave,
>>>
>>> How are you getting the metadata streams? Are they actual stream objects,
>>> or files, or database dumps, etc?
>>>
>>> As for the tools, I have used a number of the ones you listed below. I
>>> personally prefer JIRA (and it is free for non-profit). If you are ok if
>>> editing in wiki syntax I would recommend mediaWiki (it is what powers
>>> Wikipedia). You could also take a look at continuous deployment
>>> technologies like Virtual Machines (virtualbox), linux containers
>>> (docker),
>>> and rapid deployment tools (ansible, salt). Of course if you are doing
>>> lots
>>> of code changes you will want to test all of this continually (Jenkins).
>>>
>>> John Scancella
>>> Library of Congress, OSI
>>>
>>> -----Original Message-----
>>> From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of
>>> davesgonechina
>>> Sent: Tuesday, March 10, 2015 6:05 AM
>>> To: [log in to unmask]
>>> Subject: [CODE4LIB] Data Lifecycle Tracking & Documentation Tools
>>>
>>> Hi all,
>>>
>>> One of my projects involves harvesting, cleaning and transforming steady
>>> streams of metadata from numerous publishers. It's an infinite loop but
>>> every cycle can be a little bit or significantly different. Many issue
>>> tracking tools are designed for a linear progression that ends in
>>> deployment, not a circular workflow, and I've not hit upon a tool or use
>>> strategy that really fits.
>>>
>>> The best illustration I've found so far of the type of workflow I'm
>>> talking about is the DCC Curation Lifecycle Model <
>>> http://www.dcc.ac.uk/sites/default/files/documents/
>>> publications/DCCLifecycle.pdf
>>>
>>>>
>>>>  .
>>>
>>> Here are some things I've tried or thought about trying:
>>>
>>>    - Git comments
>>>    - Github Issues
>>>    - MySQL comments
>>>    - Bash script logs
>>>    - JIRA
>>>    - Trac
>>>    - Trello
>>>    - Wiki
>>>    - Unfuddle
>>>    - Redmine
>>>    - Zendesk
>>>    - Request Tracker
>>>    - Basecamp
>>>    - Asana
>>>
>>> Thoughts?
>>>
>>> Dave
>>>
>>>
>>