Carmen,

The following code may be able to help.

https://github.com/Georgetown-University-Libraries/File-Analyzer

This application can scan a file system and report counts of files by type.

The application can also report on files by checksum.  If you are trying to
find exact file duplicates, the checksum report will identify exact
duplicates found across a file system.

I will be presenting an overview of this application during the virtual
lightning talks session on April 3.

If this looks useful to you, I will be glad to give you an overview of the
application.

Terry


On Tue, Mar 19, 2013 at 4:51 PM, Carmen Mitchell
<[log in to unmask]>wrote:

> Hello Code4Libbers,
>
> I'm working with a faculty member and trying to help them to formalize
> their data collection practices. Part of this process is also going through
> old data and trying to assess what they currently have. This particular
> faculty member has been doing research for 10 years without any kind of
> structure or regular method. So far we have over 2 TB of data in various
> states. (With more to come.)
>
> I've got a programmer working with me to:
> a) identify file types
> b) count how many files of each type
>
> We are now working on de-duping and assessing file size, focusing on the
> JPEGs first. With over 300,000 over them...it might take a while. (Of
> course they aren't following any kind of file naming structure,
> either...It's a mess.)
>
> Any tips or tricks or tools that you might know of to help speed up this
> process? Is there a good image recognition tool that you could suggest that
> would help us with automation?
>
>  Thanks,
>
> Carmen Mitchell
> Institutional Repository Librarian
> Cal State San Marcos
>



-- 
Terry Brady
Applications Programmer Analyst
Lauinger Information Technology
202-687-7053