Norsk English
Back to catalogue
N-gram

N-grams from NBdigital 2022

This resource contains n-grams - i.e. uni-, bi- and trigrams - from all books and newspapers that had been digitized at the National Library of Norway up to July 15 2022. The n-grams have been extracted from a material consisting of approximately 610,000 books and 4,000,000 newspapers, amounting to a total of 138.5 billion tokens (words and punctuation). The file format is UTF-8-encoded CSV. Columns in the n-gram CSV files: - first - the first word (in uni-, bi- and trigrams) - second - the second word (in bi- and trigrams) - third - the third word (in trigrams) - lang - the language of the n-gram (only for books, the newspapers have no language classification as yet) - freq - the total frequency of the n-gram in the collection of books and newspapers - json - a dictionary with raw frequency for each year totals.json contains aggregated frequencies per year in the book and newspaper corpora. Using them, relative frequencies can be calculated in order to compare frequencies over time as in NB N-gram. metadata-digibok.csv and metadata-digavis.csv contain simple metadata for the books and newspapers. More extensive metadata can be obtained through Oria or the APIs at https://api.nb.no/. See the documentation files for further information.

Documentation

N-grams from NBdigital 2021

Description

This resource contains n-grams - i.e. unigrams, bigrams and trigrams - from all books and newspapers that had been digitized at the National Library of Norway up to July 2022. The n-grams have been extracted from a material consisting of approximately 610,000 books and 4,000,000 newspapers. The n-grams are offered as CSV files (utf8-encoded).

Contents

Columns in the n-gram CSV files: first - the first word (in unigrams, bigrams and trigrams) second - the second word (in bigrams and trigrams) third - the third word (in trigrams) lang - the language of the n-gram (only in books, newspapers have no language classification as for now) freq - the total frequency of the n-gram in the collection of books or newspapers json - a dictionary with raw frequency for each year

totals.json contains aggregated frequencies per year in the book and newspaper corpora. Using them, you can calculate relative frequencies in order to compare frequencies over time as in NB N-gram.

metadata-digibok.csv and metadata-digavis.csv contain simple metadata for the books and newspapers. If you need more extensive metadata, you could use Oria or the APIs at https://api.nb.no/.

Data preparation

The n-grams were extracted from the fulltext databases provided by the DH-lab at the National Library of Norway (https://www.nb.no/dh-lab/). The following frequency cuts were applied:

  • unigram (books): total frequency of less than five in the corpus for Norwegian Bokmål, less than two in all other languages. Additionally, the ngram must appear in more than one year.
  • unigram (newspapers): total frequency of less than five in the corpus. Additionally, the ngram must appear in more than one year.
  • bigram (books and newspapers): both parts of the bigram must appear as a unigram after the cuts above were applied. Apart from that, the same rules as for the unigrams were applied.
  • trigram (books and newspapers): the two bigrams of the trigram must appear as a bigram after the cuts above were applied. Additionally, hapax trigrams within one year were removed. Apart from that, the same rules as for the unigrams were applied.

License

The data are released in the public domain (CC-0).

Downloads