DUKweb, diachronic word representations from the UK Web Archive corpus

Lexical semantic change (detecting shifts in the meaning and usage of words) is an important task for social and cultural studies as well as for Natural Language Processing applications. Diachronic word embeddings (time-sensitive vector representations of words that preserve their meaning) have become the standard resource for this task. However, given the significant computational resources needed for their generation, very few resources exist that make diachronic word embeddings available to the scientific community. In this paper we present DUKweb, a set of large-scale resources designed for the diachronic analysis of contemporary English. DUKweb was created from the JISC UK Web Domain Dataset (1996–2013), a very large archive which collects resources from the Internet Archive that were hosted on domains ending in ‘.uk’. DUKweb consists of a series word co-occurrence matrices and two types of word embeddings for each year in the JISC UK Web Domain dataset. We show the reuse potential of DUKweb and its quality standards via a case study on word meaning change detection. Measurement(s) Natural Language • lexical semantic change Technology Type(s) Programming Language • word embeddings • Cosine Distance Method Factor Type(s) time period Measurement(s) Natural Language • lexical semantic change Technology Type(s) Programming Language • word embeddings • Cosine Distance Method Factor Type(s) time period Machine-accessible metadata file describing the reported data: https://doi.org/10.6084/m9.figshare.16438863

1. co-occurrences matrices for each year built by relying on the original text extracted from the JISC UK Web Domain Dataset; 2. a set of different word embeddings (Temporal Random Indexing and word2vec) for each year; 3. time series of words' change in representation across time.
DUKweb can be used for several time-independent NLP tasks, including word similarity, relatedness, analogy, but also for temporally dependent tasks, such as semantic change detection (i.e., tracking change in word meaning over time), a task which has received significant attention in recent years 22,23 .
The main innovative features of our contribution are: • Variety of object types: we release the full package of data needed for diachronic linguistic analyses of word meaning: co-occurrence matrices, word embeddings, and time series. • Size: we release word vectors trained on a very large contemporary English diachronic corpus of 1,316 billion word occurrences spanning the years 1996-2013; the total size of the dataset is 330GB; • Time dimension: the word vectors have been trained on yearly portions of the UK web archive corpus corpus, which makes them ideally suited for many diachronic linguistic analyses; • Embedding types: we release count-based and prediction-based types of word embeddings: Temporal Random Indexing vectors and word2vec vectors, respectively, and provide the first systematic comparison between the two approaches; None of the other existing datasets offer researchers all the above features. The surveyed datasets are based on corpora smaller than the UK Web archive JISC dataset, the biggest one being 850 billion words vs. 1316 billion words 11 . Moreover, only the Twitter embedding resource 20 was specifically built to model recent language change (2012-2018). On the other hand, recent workshops on semantic change detection 22,23 study the semantic change across the two distinct time periods and therefore lack the longitudinal aspect of our resources. In addition to being based on a much larger corpus with a longitudinal component, DUKweb can be readily used to study semantic change in English between 1996 and 2013 and therefore to investigate the effects of various phenomena such as the expansion of the World Wide Web or social media on the English language. Finally, DUKweb offers researchers a variety of object types and embedding types; as we show in our experiments, these capture the semantic change of different words and can therefore be leveraged in conjunction for further analysis in future work.

Methods
The flow chart in Fig. 1 illustrates the process from corpus creation to the construction of DUKweb. In the next sub-sections we provide details about how the three parts of the dataset were built. Source data. We used the JISC UK Web Domain Dataset (1996Dataset ( -2013 24 , which collects resources from the Internet Archive (IA) that were hosted on domains ending in '.uk' , and those that are required in order to render '.uk' pages. The JISC dataset contains resources crawled by the IA Web Group for different archiving partners, the Web Wide crawls and other miscellaneous crawls run by IA, as well as data donations from Alexa (https://www. alexa.com) and other companies or institutions, therefore we do not have access to all the crawling configurations used by the different partners. The dataset contains not only HTML pages and textual resources, but also video, images and other types of files.
The JISC dataset is composed of two parts: the first part contains resources from 1996 to 2010 for a total size of 32TB; the second part contains resources from 2011-2013 for a total size of 30TB. The JISC dataset cannot be made generally available, but can be used to generate derived datasets (like DUKweb). text extraction and pre-processing. The first step in the creation of DUKweb consisted in processing the JISC web archive in order to extract its textual resources. For this purpose, we extracted the text from resources such as TXT files and parsed HTML pages. We used the jsoup library (https://jsoup.org/) for parsing HTML pages. The original JISC dataset contains files in the ARC and WARC formats, standard formats used by IA for storing data crawled from the web as sequences of content blocks. The WARC format (https://commoncrawl. org/2014/04/navigating-the-warc-file-format/, last accessed 27/11/2020) is an enhancement of the ARC format supporting a range of features including metadata and duplicate events. We converted the ARC and WARC archives into the WET format, a standard format for storing plain text extracted from ARC/WARC archives. The output of this process provided about 5.5TB of compressed WET archives.
The second step consisted in tokenizing the WET archives. For this purpose, we used the StandardAnalyzer (https://lucene.apache.org/core/7_3_1/core/index.html), last accessed 27/11/2020. provided by the Apache Lucene 4.10.4 API (https://lucene.apache.org/core/, last accessed 27/11/2020). This analyzer also provides a standard list of English stop words. The size of the tokenized corpus is approximately 3TB, with a vocabulary size of 29 million tokens and about 1200 billion occurrences. We did not apply any further text processing steps such us lemmatization or stemming because our aim was to build a language independent pipeline.
Co-occurrence matrices. The co-occurrence matrices of DUKweb store co-occurrence information for each word token in the JISC dataset processed as described in section 'Text extraction and pre-processing' . For the construction of co-occurrence matrices, we focused on the 1,000,000 most frequent words.
In order to track temporal information, we built a co-occurrence matrix for each year from 1996 to 2013. Each matrix is stored in a compressed text format, with one row for each token, where each row contains the token and the list of tokens co-occurring with it. Following standard practice in NLP, we extracted co-occurrence counts by taking into account a window of five words to the left and five words to the right of the target word 25-27 . Word embeddings. We constructed semantic representations of the words occurring in the processed JISC dataset by training word embeddings for each year using two approaches: Temporal Random Indexing and the word2vec algorithm (i.e., skip-gram with negative sampling). The next subsections provide details of each approach.
Temporal random indexing. The first set of word embeddings of DUKweb was trained using Temporal Random Indexing (TRI) [28][29][30] . We further developed the TRI approach in three directions: 1) we improved the system to make it possible to process very large datasets like the JISC UK Web Domain Dataset; 2) we introduced a new way to weigh terms in order to reduce the impact of very frequent tokens; 3) compared to our previous work on the same topic 30 , we proposed a new "cumulative" approach for building word vectors.
The idea behind TRI is to build different word spaces for each time period under investigation. The peculiarity of TRI is that word vectors over different time periods are directly comparable because they are built using the same random vectors. TRI works as follows: 1. Given a corpus C of documents and a vocabulary V of terms (V contains the terms that we want to analyse, typically, the top n frequent terms) extracted from C, the method assigns a random vector r i to each term t i ∈ V. A random vector is a vector that has values in the set {−1, 0, 1} and is sparse, with few non-zero elements randomly distributed along its dimensions. The sets of random vectors assigned to all terms in V are near-orthogonal;  Fig. 1 Flowchart of the creation of DUKweb.
2. The corpus C is split into different time periods T k using temporal information, for example the year of publication; 3. For each period T k , a word space WS k is built. Each of the terms of V occurring in T k is represented by a semantic vector. The semantic vector sv i k for the i-th term in T k is built as the sum of all the random vectors of the terms co-occurring with t i in T k . Unlike the approach proposed in 30 , the sv i k is not initialized as a zero vector, but as the the semantic vectors − sv i k 1 built in the previous period. Using this approach we are able to collect semantic features of the term across time. If the − sv i k 1 is not available, (the − sv i k 1 is not available when the term T i appears for the first time in T k ) the zero vector is used. When computing the sum, we apply some weighting to the random vector. To reduce the impact of very frequent terms, we use the weights where C k is the total number of occurrences in T k and t # i k is the number of occurrences of the term T i in T k . The parameter th is set to 0.001.
This way, the semantic vectors across all time periods are comparable since they are the sum of the same random vectors.
Time series. For each term t i DUKweb also contains a time series Γ(t i ), which can be used to track a word's meaning change over time. The time series are sequences of values, one value for each time period, and represent the semantic shift of that term in the given period. We adopt several strategies for building the time series. The baseline approach is based on term log-frequency, where each value in the series is defined as In addition to the baseline, we devised two other strategies for building the time series: . This way, we capture semantic change between two time periods; cumulative: we build a cumulative vector and compute the cosine similarity of this cumulative vector and the vector sv i k . The idea behind this approach is that the semantics of a word at point k-1 depends on the semantics of the word in all the previous time periods. The cumulative vector is the vector sum of all the previous word vectors 31 .
Skip-gram with negative sampling. The second approach we followed for generating word representations is based on the of skip-gram with negative sampling (SGNS) algorithm 32 . The skip-gram model is a two-layer neural network that aims at predicting the words (context) surrounding a particular word in a sentence. The training process is performed on a large corpus, where samples of {context, word} pairs are drawn by sliding a window of N words at a time. The resulting word vectors can appropriately represent each word based on the context in which it appears so that the distance between similar words is small and the analogy of word pairs like (king, queen) and (uncle, aunt) is maintained. Over the past few years SGNS has been widely employed and its efficiency has been demonstrated in several studies on semantic change 10,22,27,33,34 .
We split the pre-processed corpus into yearly bins, as in the case of TRI, and train one language model per year (we refrain from using the years 1996-1999 for SGNS, due to their considerably smaller size compared to the rest of the years in our processed collection, which could result into noisy word representations). Our skip-gram model then learns a single 100-dimensional representation for each word that is present at least 1,000 times on each year independently, i.e. 3,910,329 words. We used the implementation of skip-gram with negative sampling as provided in gensim (https://radimrehurek.com/gensim/), using 5 words as our window size and training for 5 epochs for each year while keeping the rest of hyperparameters on their default values. (In previous work 27 , we selected the 47.8 K words occurring in all years in both our corpus and in the entry list of the Oxford English Dictionary. Importantly, very common words (e.g., "facebook") that appeared only after a certain point in time are not included in our previous analysis.) Orthogonal procrustesIn contrast to TRI, a drawback of training independent (i.e., one per year) SGNS models is the fact that the resulting word vectors are not necessarily aligned to the same coordinate axes across different years 10 . In particular, SGNS models may result in arbitrary linear transformations, which do not affect pairwise cosine-similarities within-years, but prevent meaningful comparison across years.
To align the semantic spaces, we follow the Procrustes analysis from 10 . Denote by W (t) ∈R n × m the matrix of word embeddings in year t. The orthogonal Procrustes problem 35 consists of solving m m is the rotation matrix and I is the m × m identity matrix. We align the word embedding in year t to their respective embeddings in year t + 1 by finding a translation, rotation, and scaling of W (t) that minimizes its distance to W (t + 1) as measured by the Frobenius norm. The optimisation problem in (1) can be solved using the singular value decomposition of W (t) (W (t + 1) ) T . We can then use the cosine distance between the vector representation of a word across aligned embeddings as a measure of the semantic displacement of that word.
An alternative approach would be to initialise the embeddings of the year t + 1 with the resulting representations of the year t 15 . However, this would demand sequential -and thus longer -processing of our data and it is not clear whether the resulting representations capture the semantics of the words more effectively, as demonstrated in recent work on semantic change detection 21 . Another potential approach to consider stems from the www.nature.com/scientificdata www.nature.com/scientificdata/ generalized orthogonal Procrustes problem which simultaneously considers all the available embeddings at once and aims to optimally identify an average embedding which is simultaneously close to all the input embeddings 36 . This contrasts with our approach where only pairwise Procrustes alignments are performed.
Time series. We construct the time series of a word's similarity with itself over time, by measuring the cosine distance of its aligned representation in a certain year (2001-2013) from its representation in the year 2000. Recent work 21 has demonstrated that the year used for reference matters (i.e., W (t) in Eq. 1). We intuitively opted to use the year 2000 as our referencing point, since it is the beginning of our time period under examination. In order to construct time series in a consistent manner, we only use the representations of the words that are present in each year.

Data records
This section describes each data record associated with our dataset, which is available on the British Library repository (https://doi.org/10.23636/1209) 37 .
Co-occurrences matrices. The first part of our dataset consists of the co-occurrences matrices. We built a co-occurrence matrix from the pre-processed JISC dataset for each year from 1996 to 2013. Each matrix is stored in a compressed text format, with one row per token. Each row reports a token and the list of tokens co-occurring with it. An example for the word linux is reported in Fig. 2, which shows that the token swapping co-occurs 4 times with linux, the word google 173 times, and so on. Table 1 reports the size of the vocabulary and the associated compressed file size, for each year. The total number of tokens considering only the terms in our vocabulary is 1,316 billion.
Word embeddings. The second part of the dataset contains word embeddings built using TRI and word-2vec. Both embeddings are provided in the GZIP compressed textual format, with one file for each year. Each file stores a word embedding for each line, the line starts with a word followed by the corresponding embedding vector entries separated by spaces, for example: dog 0.0019510963-0.033144157 0.033734962…  www.nature.com/scientificdata www.nature.com/scientificdata/ Table 2 shows the (compressed) file size of each vector space. For TRI, the number of vectors (terms) for each year is equal to the vocabulary size of co-occurrences matrices as reported in Table 1. The TRI vector dimension is equal to 1,000, while the number of no-zero elements in random vector is set to 10. Finally, the parameter th is set to 0.001, and TRI vectors are built by using the code reported in section 'Code Availability' . For SGNS, the total number of words represented in any year is 3,910,329. Finally, the chart in Fig. 3 shows the intersected vocabulary between the two methods. The number of total terms contained in the intersection is 47,871. We also release a version of TRI embeddings built by taking into account only the terms contained in the intersection. In order to perform a fair evaluation, in our experiments we only take into account the intersected vocabulary. time series. The last part of the dataset contains a set of time series in CSV format computed using the different strategies described in the previous sections. For each time series we extract a list of change points. The chart on Fig. 4(a) shows the time series for different words that have acquired a new meaning after the year 2000, according to the Oxford English Dictionary. Similarly, Fig. 4(b) shows the time series of the cosine distances of the same words that result after Orthogonal Procrustes is applied on SGNS as described in Section 'Skip-gram with Negative Sampling' . In particular, we first align the embeddings matrices of a year T with respect to the year   Figure 4 shows that the semantic similarity of the word "blackberry" decreases dramatically in 2004, which corresponds to the change point year detected by the semantic change detection algorithm. On the other hand, Fig. 4(b) shows that the four words are moving away with time from their semantic representation in the year 2000. We also find examples of cases of semantic shifts corresponding to real-world events in SGNS representations: for example, the meaning of the word "tweet" shifted more rapidly after Twitter's foundation (2006), whereas the lexical semantics of the word "tablet" mostly shifted in the year 2010, at the time when the first mass-market tablet -iPad -was released.
Furthermore, as we showed in our previous work 30 , it is possible to analyze the neighborhood of a word (e.g.

technical Validation
We perform two sets of experiments in order to assess the quality of the embeddings generated via TRI and SGNS. In the first set, our goal is to measure the embeddings' ability to capture semantic properties of words, i.e. analogies, similarities and relatedness levels. In the second set, we aim at exploring to what extent the two types of contextual vectors capture the change in meaning of words.

Static tasks: Word-level Semantics.
In this set of tasks we examine the ability of the word vectors to capture the semantics of the words associated to them. We work on three different sub-tasks: (a) word analogy, (b) word similarity and (c) word relatedness detection. Tasks description. Word analogy. Word analogy detection is the task of identifying relationships between words in the form of "w a is to w b as w c is to w d ", where w i is a word. In our experiments, we make use of the widely employed dataset which was created by Mikolov and colleagues 32 and which contains these relationships in different categories (we list four categories, with one example for each of them): • Geography, e.g. Paris is to France as Madrid is to Spain. www.nature.com/scientificdata www.nature.com/scientificdata/ • Currencies, e.g. Peso is to Argentina as euro is to Europe. • Family, e.g. Boy is to girl as policeman is to policewoman. • Grammar rules, e.g. Amazing is to amazingly as apparent is to apparently. TRI-based embeddings are not suitable for this kind of task due to the nature of the vector space. TRI is not able to learn the linear dependency between vectors in the space, which may be due to the Random Projection that preserves all distance/similarity measures based on L2-norm, but it distorts the original space and does not preserve the original position of vectors. We try to simulate analogy by using vector orthogonalization as the negation operator 38 . In particular, the vector sum v b + v d is orthogonalized with respect to the vector v a . For each word vector the cosine similarity is computed against the vector obtained as the result of the orthogonalization, then the word with the highest similarity is selected.
We perform experiments using our TRI and SNGS word vectors. For comparison purposes, we also employ the word2vec pre-trained embeddings generated in 32 as well as the GloVe embeddings released in 39 . These are well-established sources of word vectors that have been trained on massive corpora of textual data and have been employed in an extremely large body of research work across multiple NLP tasks. In particular, for word2vec we employ the 300-dimensional word vectors that were trained on Google news (pre w2v ) 32 , whereas for GloVe we use the 100-dimensional vectors trained on Twitter (pre glv ) 39 . As opposed to SGNS and TRI, pre w2v and pre glv are temporally independent (i.e., there is one representation of each word through time). To allow for a fair comparison, the analysis described here and in the following sections is on the intersected vocabulary across all years, i.e. on the set of words occurring in all years in our corpus. Word similarity. In the word similarity task, we are interested in detecting the similarity between two words. We employ the dataset in 40 , which contains examples of 203 word pairs, along with their similarity score, as provided by two annotators. We only use the 187 word pairs that consist of words present in the intersected vocabulary, as before. We deploy the following experimental setting: given the word vectors w a and w b of a pair of words, we measure their cosine similarity sim(w a , w b ); subsequently, we measure the correlation between sim(w a , w b ) and the ground truth. Higher values indicate a better model. We compare performance against the baselines pre w2v and pre glv . Word relatedness. Our final experiment in this section involves the task of identifying the level of relatedness between two words. For example, the word "computer" is much more closely related to the word "software" than to the word "news". We employ the word relatedness dataset introduced by 40 , which contains 252 examples of word pairs, along with their associated relatedness score. As in the previous tasks, we only use the 232 examples that are present in our intersected vocabulary; we deploy the same experimental setting as in the case of the Word Similarity task and compare against the two previously introduced baselines (pre w2v , pre glv ).

results
Word analogy. The results are displayed in Fig. 5. pre glv outperforms our models in almost all cases. SGNS achieves comparable performance to pre glv and pre w2v , especially during the first years; comparably. TRI performs poorly since it is not able to learn the linear dependency between vectors in the space, which may be due to the Random Projection that preserves all distance/similarity measures based on L2-norm, but it distorts the original www.nature.com/scientificdata www.nature.com/scientificdata/ space and does not preserve the original position of vectors. pre w2v , pre glv and SGNS better capture relationships related to "Family" and "Grammar" than "Currency" in our experiments.
Word relatedness and word similarity. Figure 6 shows the results on the word similarity and word relatedness tasks. Here pre w2v achieves the highest average Pearson correlation score across all years (0.76 and 0.58 for the two tasks, respectively). SGNS performs competitively (0.57 vs 0.61, on average) and outperforms in most cases the pre glv baseline for the case of semantic similarity and relatedness, respectively. Importantly, its performance is again consistent across time, ranging from 0.51 to 0.63 and from 0.40 to 0.56 for the two respective tasks. TRI performs again more poorly and slightly more inconsistently than SGNS, with its (across years) evaluation score ranging from 0.22 to 0.38 (average: 0.28) and from 0.08 to 0.29 (average: 0.22).
The results presented so far show that, overall, the SGNS and TRI embeddings do not outperform the two baselines (pre w2v and pre glv ) consisting of static word representations in temporally independent tasks. This is partially attributed to the facts that our resources are built on large-scale, yet potentially noisy content and are also restricted primarily to British English, which could impose some geographical and linguistic biases. However, both representations (SGNS, TRI) can be effectively utilised for a dynamic, i.e. temporally dependent, task which cannot be dealt with static word representations, as we show in the next section.
Dynamic task: Semantic change detection. The major contribution of the TRI and SGNS embeddings of DUKweb consists in their temporal dimension. To exploit this aspect, we measure their ability to capture lexical semantic change over time.
Experimental setting. We use as ground truth 65 words that have changed their meaning between 2001 and 2013 according to the Oxford English Dictionary 27 . We define the task in a time sensitive manner: given the word representations in the year 2000 and in the year X, our aim is to find the words whose lexical semantics have changed the most. We vary X from 2001 to 2013, so that we get clearer insights on the effectiveness of the word representations. Since our ground truth consists of words that are present in the OED, we also limit our analysis to the 47,834 words that are present both in the OED and in the TRI/SGNS vocabularies.

Models.
We employ four variants of Orthogonal Procrustes-based methods operating on SGNS from our prior work 27,41 and two models operating on TRI, as follows: • SGNS pr employs the Orthogonal Procrustes (OP) alignment to align the word representations in the year X based on those for the year 2000; • SGNS pr(a) applies OP in two passes: during the first pass, it selects the most stable ("anchor") words (i.e., those whose aligned representations across years have the highest cosine similarity); then, it learns the alignment between the representations in the years 2000 and X using solely the anchor words; • SGNS pr(d) applies OP in several passes executed in two rounds: during the first round, it selects the most stable ("diachronic anchor") words across the full time interval; then, it learns the alignment between the representations in the years 2000 and X using solely the diahcronic anchor words; • TRI c and TRI p exploit time-series built by the cumulative and point-wise approach, respectively (see Section 'Temporal Random Indexing').
Evaluation. In all of the SGNS models, we rank the words on the basis of the cosine similarity of their representations in the aligned space, such that words in the higher ranks are those whose lexical semantics has www.nature.com/scientificdata www.nature.com/scientificdata/ changed the most. For the TRI-based models, the Mean Shift algorithm 42 is used for detecting change points in the time series consisting of the cosine similarity scores between the representations of the same word in each year covered by the corpus. For each detected change point, a p-value is estimated according to the confidence level obtained by the bootstrapping 43 approach proposed in 42 , then words are ranked according to the p-value in ascending order. Finally, we run the evaluation using the recall at 10% of the size of the dataset as well as the average rank (μ r , scaled in [0, 1]) of the 65 words with altered semantics. Higher recall-at-k and lower μ r scores indicate a better performance.
Results. Tables 4 and 5 present the results of our models in terms of μ r and recall-at-k, respectively. In both cases, the SGNS-based approaches perform better than TRI: on average, the best-performing SGNS-based model achieves 27.42 in μ r (SGNS pr(a) ) and 29.59 in recall-at-k (SGNS pr ). The difference compared to TRI is attributed to their ability to better capture the contextualised representation of the words in our corpus. Nevertheless, TRI has recently achieved state-of-the-art performance on semantic change detection in the Swedish language 22 . Furthermore, despite their superior performance in this task, the Procrustes-and SGNS-based approaches have the shortcoming that they operate on a common vocabulary across different years; thus, words that have appeared at a certain year cannot be detected in these variants -a drawback is not present in the case of TRI.
Finally, we inspect the semantically altered words that have been detected by each model. Table 3 displays the most "obvious" and challenging examples of semantic change, as ranked on a per-model basis. It becomes evident that the two different word representations better capture the changes of different words. This is attributed to the different nature of the two released word representations. Incorporating hybrid approaches operating on multiple embedding models could be an important direction for future work in this task.

Usage Notes
The DUKweb datasets can be used for various time-independent tasks, as demonstrated in this article. Their major application is for studying how word meaning changes over time (i.e., semantic change) in a computational and linguistic context. The instructions on how to run our code for the experiments as well as for further downstream tasks have been made publicly available (see Code Availability). Access to the original JISC UK    www.nature.com/scientificdata www.nature.com/scientificdata/ Web Domain Dataset (1996-2013) collection 24 may be sought by contacting the UK Web Archive (https://doi. org/10.5259/ukwa.ds.2/1).

Code availability
The creation of the described datasets requires several steps, each step is performed by a different software. All the software is freely available, in particular: • the code for the processing of the JISC UK Web Domain Dataset for producing both the WET and tokenized files: https://github.com/alan-turing-institute/UKWebArchive_semantic_change; • the software for building both co-occurrences matrices and TRI: https://github.com/alan-turing-institute/ temporal-random-indexing; • th e code for the experiments can be found at https://github.com/alan-turing-institute/DUKweb; For information on our input data, refer to: https://data.webarchive.org.uk/opendata/ukwa.ds.2/.