I was involved in some work on calculating and visualising this kind of
word/phrase frequency on a project a few years ago - this was based on
queries to a corpus indexed in ElasticSearch, and charting words/phrase
frequency against each other using javascript - see
https://ukmhl.historicaltexts.jisc.ac.uk/ngram for an example - but
although it looks superficially similar it isn't anywhere close to as
sophisticated as the Google n-gram viewer.
You probably already know, but I think it's worth stating, that the
Google n-gram viewer (https://books.google.com/ngrams) is not simply
visualising the frequency of word/phrase occurrences, but the frequency
as a percentage of the frequency of all n-grams of the same size in the
corpus https://books.google.com/ngrams/info. The Google n-gram viewer
goes well beyond this as well, supporting ways of being more specific
(e.g. you can limit by part of speech, and by language of the texts
analysed). This suggests a sophisticated linguistic parsing of a large
corpus with the ability to answer complex questions quickly at a scale -
something we weren't able to do in our project.
In the project I was involved in, we are simply showing the percentage
of texts in the corpus in which a word appears, not the frequency as a
percentage of all same sized n-grams - which means our viewer is more
about reflecting general book topics than it is about linguistic
analysis. You can also see issues with the measurement at either end of
the graph where there seem to be spikes in usage - but this actually
reflects that the collection simply lacks large numbers of texts from
those years, which means a term only has to appear in a few books to get
a high percentage. Despite this, the tool is still useful (IMO) within
those constraints - in the example search it is possible to see that the
term "tubercolosis" rises in frequency, while the term "phthisis" (for
the same condition) drops off - a trend also shown by Google n-gram
viewer
https://books.google.com/ngrams/graph?content=tuberculosis%2Cphthisis&year_start=1800&year_end=1930&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Ctuberculosis%3B%2Cc0%3B.t1%3B%2Cphthisis%3B%2Cc0
Finally if you want to do more sophisticated analysis it may be worth
looking at specialist tools - e.g. AntConc
http://www.laurenceanthony.net/software/antconc/
Hope some of that is helpful
Owen
Fitchett, Deborah wrote on 08/11/2019 01:10:
> You might be interested in Chart.js (https://www.chartjs.org/) - it does the visualisation part, if you could do the search part.
>
> Deborah
>
> -----Original Message-----
> From: Code for Libraries <[log in to unmask]> On Behalf Of Vinit Kumar
> Sent: Thursday, 7 November 2019 6:17 PM
> To: [log in to unmask]
> Subject: [CODE4LIB] n-gram visualisation
>
> Dear Code4Libers,
>
> I have data with the following structure:
> ngrams 2009 2010 2011 2012 2013 2014 2015
> library 22 3 32 32 35 21 21
> technology 3 4 43 32 30 43 32
> and so on
>
> Is it possible to visualise this data in a similar manner as Google N-gram viewer displays? Wherein one can put the keyword in a search bar and the visualisation displays the year wise trend of that keyword in the corpus based on the above structured data.
> Any pointers or tools would be of help.
> Thanking you in anticipation.
>
>
> --
> Regards
> Vinit Kumar, Ph.D.
> Assistant Professor,
> Department of Library and Information Science Babasaheb Bhimrao Ambedkar University, Rae Bareilly Road, Lucknow, India 226025
> +919454120174
>
>
> ________________________________
>
> "The contents of this e-mail (including any attachments) may be confidential and/or subject to copyright. Any unauthorised use, distribution, or copying of the contents is expressly prohibited. If you have received this e-mail in error, please advise the sender by return e-mail or telephone and then delete this e-mail together with all attachments from your system."
--
Sent from Postbox <https://www.postbox-inc.com>
|
