On Sep 27, 2017, at 8:18 AM, Eric Lease Morgan <[log in to unmask]> wrote:
>>>>> Does anybody here know how to access a Python compressed sparse row format (CSR) object? -> http://bit.ly/2fPj42V
>>>>
>>>> Do you have a link to the code you're using?
>>>
>>> Yes, thank you. See —> http://dh.crc.nd.edu/sandbox/htrc-workset-browser/bin/topic-model.py —ELM
>>
>> I'm not familiar with the APIs in question, but--if I'm looking at this right, your CSR matrix (tfidf) looks like it would have columns corresponding with topics and rows corresponding with documents...
>>
>> Jason, this is REALLY close, and I have begun to include it at the very end of my code. Thank you! ‘More later.
Jason’s suggestions were very helpful, and after hacking on my topic modeling program I am giving the program a label of version 1.0. See -> http://dh.crc.nd.edu/sandbox/htrc-workset-browser/bin/topic-model.py
To resolve my issues I had to:
1. learn that my vectorizer (TfidfVectorizer) could take
a list of file names as input, thus the file names
come along for the ride in the resulting matrices
2. exploit Jason’s suggestion to extract the file names
from a list of ranked (sorted) topics
If you have a directory of plain text files, and if you have Python (as well as all of its friends) installed, then you can run the program something like this:
$ ./topic-model.py ./shamanism/text/ 5 5 3
The result will be a list of topics (think “subject terms”) and their most associated files:
* god; church; christian; divine; christ
o ./shamanism/text/uva.x002756372.txt
o ./shamanism/text/uc1.$b43226.txt
o ./shamanism/text/mdp.39015062241685.txt
* god; gods; primitive; spirits; worship
o ./shamanism/text/hvd.ah59xi.txt
o ./shamanism/text/uc2.ark+=13960=t3mw2kc4t.txt
o ./shamanism/text/mdp.39015025016869.txt
* social; cultural; culture; group; economic
o ./shamanism/text/uc1.b4558415.txt
o ./shamanism/text/uc1.$b604512.txt
o ./shamanism/text/mdp.39015003464057.txt
* russian; siberia; river; south; russia
o ./shamanism/text/uc2.ark+=13960=t1qf8s666.txt
o ./shamanism/text/mdp.39015068416885.txt
o ./shamanism/text/njp.32101068979754.txt
* cf; hebrew; el; text; babylonian
o ./shamanism/text/nyp.33433081840559.txt
o ./shamanism/text/wu.89097203632.txt
o ./shamanism/text/uc2.ark+=13960=t84j0cs6h.txt
What’s really cool is that I can now search my corpus for terms like god, church, or Christian, and the ranked files float to the top; this particular topic modeling process works for me, and now I can “turn the knobs” to improve the results as well as consider plotting the results on a Cartesian plane to visualize similarity.
Fun with text mining.
—
Eric Lease Morgan
University of Notre Dame
