Norsk English
Back to catalogue
N-gram

N-grams from NBdigital 2021

This resource contains n-grams - i.e. unigrams, bigrams and trigrams - from all books and newspapers that had been digitized at the National Library of Norway up to July 2021. The n-grams have been extracted from a material consisting of approximately 580,000 books and 3,400,000 newspapers, amounting to a total of 122 billion tokens (words and punctuation). The n-grams are offered as CSV files (UTF-8-encoded). Columns in the n-gram CSV files: - first - the first word (in uni-, bi- and trigrams) - second - the second word (in bi- and trigrams) - third - the third word (in trigrams) - lang - the language of the n-gram (only regarding books, newspapers have no language classification as for now) - freq - the total frequency of the n-gram in the collection of books or newspapers - json - a dictionary with raw frequency for each year totals.json contains aggregated frequencies per year in the book and newspaper corpora. Using these numbers, relative frequencies can be calculated in order to compare frequencies over time as in NB N-gram. metadata-digibok.csv and metadata-digavis.csv contain simple metadata for the books and newspapers. If you need more extensive metadata, you could use Oria or the APIs at https://api.nb.no/. See the documentation files for further information.

Documentation

N-grams from NBdigital 2021

Description

This resource contains n-grams - i.e. unigrams, bigrams and trigrams - from all books and newspapers that had been digitized at the National Library of Norway up to July 2021. The n-grams have been extracted from a material consisting of approximately 580,000 books and 3,400,000 newspapers. The n-grams are offered as CSV files (utf8-encoded).

Contents

Columns in the n-gram CSV files: first - the first word (in unigrams, bigrams and trigrams) second - the second word (in bigrams and trigrams) third - the third word (in trigrams) lang - the language of the n-gram (only in books, newspapers have no language classification as for now) freq - the total frequency of the n-gram in the collection of books or newspapers json - a dictionary with raw frequency for each year

totals.json contains aggregated frequencies per year in the book and newspaper corpora. Using them, you can calculate relative frequencies in order to compare frequencies over time as in NB N-gram.

metadata-digibok.csv and metadata-digavis.csv contain simple metadata for the books and newspapers. If you need more extensive metadata, you could use Oria or the APIs at https://api.nb.no/.

Data preparation

The n-grams were extracted from the fulltext databases provided by the DH-lab at the National Library of Norway (https://www.nb.no/dh-lab/). The following frequency cuts were applied:

  • unigram (books): total frequency of less than five in the corpus for Norwegian Bokmål, less than two in all other languages. Additionally, the ngram must appear in more than one year.
  • unigram (newspapers): total frequency of less than five in the corpus. Additionally, the ngram must appear in more than one year.
  • bigram (books and newspapers): both parts of the bigram must appear as a unigram after the cuts above were applied. Apart from that, the same rules as for the unigrams were applied.
  • trigram (books and newspapers): the two bigrams of the trigram must appear as a bigram after the cuts above were applied. Additionally, hapax trigrams within one year were removed. Apart from that, the same rules as for the unigrams were applied.

License

The data are released in the public domain (CC-0).

Downloads