The code used for overlap detection within the arXiv corpus (see [1] 
which significantly extended earlier work [2]) does a matching based on 
a sliding window of hashed 7-word sequences on extracted ASCII text. 
Perhaps more the required for the case in question, but this approach 
scales to a corpus of 1M articles. I'm afraid it is a little finicky to 
compile on current systems given all the changes in C++ library 
organization since I wrote it in 2005/2009. I'm working off-and-on to 
tidy it up but haven't got there yet... So, FWIW, code at:

https://github.com/zimeon/docsim

Cheers,
Simeon

[1] http://arxiv.org/abs/1412.2716
[2] http://arxiv.org/abs/cs/0702012

On 1/23/15 9:44 AM, Mark A. Matienzo wrote:
> I believe Turnitin and SafeAssign both compare the text of submissions to
> against external sources (e.g., SafeAssign uses ABI/INFORM, among others).
> I am not certain if they compare submissions against each other.
>
> However, if you're looking for something along the lines of what Dre
> suggests, you could use ssdeep, which is an implementation of a piecewise
> hashing algorithm [0]. The issue with that you would have to assume that
> all students would probably be using the same file format.
>
> You could also using something like Tika to extract the text content from
> all the submissions, and then compare them against each other.
>
> [0] http://ssdeep.sourceforge.net/
> [1] http://tika.apache.org/
>
> Mark
>
> --
> Mark A. Matienzo <[log in to unmask]>
> Director of Technology, Digital Public Library of America
>
> On Fri, Jan 23, 2015 at 8:47 AM, Andreas Orphanides <[log in to unmask]>
> wrote:
>
>> My first thought was something like programatically doing a pairwise diff
>> of the files, 5500 times. I was surprised I couldn't find a utility that
>> just does this.
>>
>> But i did find something called diffuse [1], that allows you to graphically
>> compare any number of text files in a diff-like fashion. This would
>> probably at least be able to tell you which files need closer scrutiny.
>>
>> I think you'd presumably have to be able to extract the text from each
>> file; I doubt it would work on raw Word docs or PDFs, so that might be a
>> stopper.
>>
>> It seems like the realm of source control has a lot of software designed to
>> help with this problem, so there might be other similar things out there.
>> But probably not anything designed to natively handle print-ready files.
>>
>> -dre.
>>
>>
>> [1] http://diffuse.sourceforge.net/about.html
>>
>> On Fri, Jan 23, 2015 at 7:26 AM, Judy Meirose <[log in to unmask]> wrote:
>>
>>> Can anyone recommend a plagiarism checking software besides Turnitin and
>>> SafeAssign?  I need to compare about 100 student assignments against each
>>> other to make sure they don't copy each other's assignments.
>>>
>>> Thanks.
>>>
>>> Judy K. Meirose
>>> Systems Librarian
>>> Florida Coastal School of Law
>>> 8787 Baypine Rd
>>> Jacksonville, FL
>>> (904)680-7603
>>>
>>> This email transmission, and any documents, files or previous e-mail
>>> messages attached to it, may contain confidential, privileged and/or
>>> proprietary information for the sole use of the intended recipient(s). If
>>> you are not an intended recipient or a person responsible for delivering
>> it
>>> to an intended recipient, any disclosure, copying, distribution or use of
>>> any of the information contained in or attached to this transmission is
>>> strictly prohibited. If you have received this transmission in error,
>>> please: (1) immediately notify me by reply e-mail; and (2) destroy the
>>> original (and any copies) of this transmission and its attachments
>> without
>>> reading or saving in any manner.
>>>
>>