A general-purpose material property data extraction pipeline from large polymer corpora using natural language processing

The ever-increasing number of materials science articles makes it hard to infer chemistry-structure-property relations from literature. We used natural language processing methods to automatically extract material property data from the abstracts of polymer literature. As a component of our pipeline, we trained MaterialsBERT, a language model, using 2.4 million materials science abstracts, which outperforms other baseline models in three out of five named entity recognition datasets. Using this pipeline, we obtained ~300,000 material property records from ~130,000 abstracts in 60 hours. The extracted data was analyzed for a diverse range of applications such as fuel cells, supercapacitors, and polymer solar cells to recover non-trivial insights. The data extracted through our pipeline is made available at polymerscholar.org which can be used to locate material property data recorded in abstracts. This work demonstrates the feasibility of an automatic pipeline that starts from published literature and ends with extracted material property information.


Introduction
The number of materials science papers published annually grows at the rate of 6 % compounded annually.Quantitative and qualitative materials property information is locked away in these publications written in natural language that is not directly machine-readable.
The explosive growth in published literature makes it harder to see quantitative trends by just directly analyzing large amounts of literature.Searching the literature for material systems that have desirable properties also becomes more challenging.Moreover, material information published in a non-machine-readable form contributes to data scarcity in the field of materials informatics where the training of property predictors requires painstaking manual curation of the data of interest from literature.Here, we propose adapting techniques for information extraction from the natural language processing (NLP) literature to address these issues.
Information extraction from the written text is well studied within NLP and involves several key components such as named entity recognition (NER), i.e., identifying categories to which words in the text belong; relation extraction, i.e., classifying relationships between extracted entities; co-referencing, i.e., identifying clusters of named entities in the text referring to the same object such as a polymer and its abbreviation, and named entity normalization, i.e. identifying all the variations in the name for an entity across a large number of documents.The idea of "self-supervised learning" through transformer-based models such as BERT, 1,2 pre-trained on massive corpora of unlabeled text to learn contextual embeddings, is the dominant paradigm of information extraction today.A common architecture for NER and relation extraction is to feed a labeled input to BERT and use the output vector embedding for each word along with the corresponding labels (which could be entity labels or relation labels) as inputs to a downstream machine learning model (typically a neural network) that learns to predict those labels.The tasks mentioned above are label intensive.
Extending these methods to new domains requires labeling new data sets with ontologies that are tailored to the specific domain of interest.
ChemDataExtractor, 3 ChemSpot, 4 and ChemicalTagger 5 are tools that perform NER to tag material entities.For example, ChemDataExtractor has been used to create a database of Neel temperatures and Curie temperatures that were automatically mined from literature. 6 has also been used to generate a literature extracted database of magnetocaloric materials and train property prediction models for key figures of merit. 7In the space of polymers, the authors of Ref. 8 used a semi-automated approach that crawled papers automatically and used students to extract the Flory-Huggins parameter (a measure of the affinity between two materials, eg., a polymer and a solvent).Word embedding approaches were used in Ref. 9 to generate entity-rich documents for human experts to annotate which were then used to train a polymer named entity tagger.Most previous NLP-based efforts in materials science have focused on inorganic materials but limited work has been done to address information extraction challenges in polymers.Polymers in practice have several non-trivial variations in name for the same material entity which requires polymer names to be normalized.Moreover, unlike inorganic entities, polymer names cannot typically be converted to SMILES strings that are usable for downstream machine learning but the SMILES strings must instead be inferred from figures in the paper that contain the corresponding structure.
Past work to automatically extract material property information from literature has focused on specific properties typically using keyword search methods or regular expressions. 10wever, there are few solutions in the literature that address building general-purpose capabilities for extracting material property information, i.e., for any material property.Moreover, property extraction and analysis of polymers from a large corpus of literature has also not yet been addressed.Automatically analyzing large materials science corpora has enabled many novel discoveries in recent years such as Ref. 11, where a literature extracted data set of zeolites was used to analyze interzeolite relations.Using word embeddings trained on such corpora has also been used to predict novel materials for certain applications in inorganics as well as polymers. 12,13 this work, we built a general-purpose pipeline for extracting material property data.
Starting with a corpus of 2.4 million materials science papers described in Ref. 12, we selected 750 abstracts from the polymer domain and annotated each of the abstracts using our own ontology that was designed for the purpose of extracting information from materials science literature.Using these 750 annotated abstracts we trained an NER model, using our MaterialsBERT language model to encode the input text into vector representations.Mate-rialsBERT in turn was trained by starting from PubMedBERT, another language model, and using 2.4 million materials science abstracts to continue training the model. 14The trained NER model was applied to polymer abstracts and heuristic rules were used to combine the predictions of the NER model and obtain material property records from all polymer relevant abstracts.This pipeline is illustrated in Figure 1.We restricted our focus to abstracts as associating property value pairs with their corresponding materials is a more tractable problem in abstracts.We analyzed the data obtained using this pipeline for applications as diverse as polymer solar cells, fuel cells, and supercapacitors and showed that several known trends and phenomena in materials science can be inferred using this data.Moreover, we trained a machine learning predictor for the glass transition temperature using automatically extracted data (Supplementary Information Section S5).This is the first work to build a general-purpose material property data extraction pipeline, for any material property.MaterialsBERT, the language model that powers our information extraction pipeline is released in order to enable the information extraction efforts of other materials researchers.We show that MaterialsBERT outperforms other similar BERT-based language models such as BioBERT 15 and ChemBERT 16 on three out of five materials science NER data sets.The data extracted using this pipeline can be explored using a convenient web-based interface (https://www.polymerscholar.org)which can aid polymer researchers in locating material property information of interest to them.

Abstract annotation
Our ontology for extracting material property information consists of 8 entity types namely POLYMER, POLYMER CLASS, PROPERTY VALUE, PROPERTY NAME, MONOMER, ORGANIC MATERIAL, INORGANIC MATERIAL, and MATERIAL AMOUNT.For a detailed description of these entity types, see Table 1.This ontology captures the key pieces of information commonly found in abstracts and the information we wish to utilize for downstream purposes.Note that, unlike some other studies, our ontology scheme does not annotate entities using the BIO tagging scheme, i.e., Beginning-Inside-Outside of the labeled entity.Instead, we opt to keep the labels simple and annotate only tokens belonging to our ontology and label all other tokens as 'OTHER'.This is because, as reported in Ref.
14, for BERT-based sequence labeling models, the advantage offered by explicit BIO tags is negligible and IO tagging schemes suffice.More detailed annotation guidelines are provided in the Supplementary Information Section S1.A corpus of 2.4 million materials science papers was filtered to obtain a dataset of abstracts that were polymer relevant and likely to contain the entity types of interest to us.We did so by filtering abstracts containing the string 'poly' to find polymer-relevant abstracts and used regular expressions to find abstracts that contained numeric information.
Using the above-described ontology, we annotated 750 abstracts and split the abstracts into 80 % for training, 10 % for validation, and 10 % for testing.Prior to manual annotation, we pre-annotated the dataset using dictionaries of entities for the entity types where one was available. 17This was intended to speed up the annotation process.This dataset was annotated by three domain experts using the tool Prodigy (https://prodi.gy).Annotation was done over three rounds using a small sample of abstracts in each round.With each round, the annotation guidelines were refined and the abstracts in the previous rounds were re-annotated using the refined guidelines.
In order to assess the inter-annotator agreement between the three annotators, we use 10 common abstracts to measure Cohen's Kappa and Fleiss Kappa 18 metrics.The Fleiss Kappa metric was computed to be 0.885 and the pairwise Cohen's Kappa metric to be (0.906, 0.864, 0.887) for each of the three pairs of annotators.These metrics are comparable to interannotator agreements reported elsewhere in the literature 19 and indicate good homogeneity in the annotations.

NER model
The architecture used for training our NER model is depicted in Figure 2. BERT and BERT-based models have become the de-facto solutions for a large number of NLP tasks. 1 It embodies the transfer learning paradigm in which a model is trained on a large amount of unlabeled text using unsupervised objectives (not shown in the figure).The resulting BERT encoder can be used to generate token embeddings for the input text that are conditioned on all other input tokens and hence are context-aware.We used a BERT-based encoder to generate representations for tokens in the input text as shown in Figure 2. The generated representations were used as inputs to a linear layer connected to a softmax non-linearity that predicted the entity type of each token.We used a number of different encoders and compared the performance of the resulting models on our dataset of annotated polymer

Evaluation methods
The performance of the NER model is evaluated using precision, recall and F1 score of the predicted entity tag compared to the ground truth labels.These are defined as below: where TP are the true positives, FP are the false positives and FN are the false negatives.
Each of the above metrics is reported as a % value.We consider a predicted label to be a true positive only when the label of a complete entity is predicted correctly.For instance, for the polymer 'polyvinyl ethylene', both 'polyvinyl' and 'ethylene' must be correctly labeled, else the entity is deemed to be predicted incorrectly.The performance of various pre-trained BERT-based language models tested for training an NER model using our annotated data set of polymer abstracts is shown in Table 2.We observe that MaterialsBERT, the model fine-tuned by us on 2.4 million materials science abstracts using PubMedBERT as the starting point, outperforms PubMedBERT as well as other language models used.This is in agreement with results previously reported where the fine-tuning of a transformer-based language model on a domain-specific corpus results in improved downstream task performance. 14Similar trends are observed across two of the four materials science data sets as reported in Table 3 and thus MaterialsBERT outperforms other pre-trained language models in three out of five materials science data sets.These NER datasets were chosen to span a range of subdomains within materials science, i.e., across organic and inorganic materials.A more detailed description of these NER datasets is provided in Supplementary Information Section S2.Note that all pre-trained encoders tested in Table 2 use the BERT-base architecture, differing in their weights and hence are comparable.MaterialsBERT outperforms PubMedBERT on all datasets except ChemD-NER, which demonstrates that fine-tuning on a domain specific corpus indeed produces a performance improvement on downstream classification tasks.ChemBERT is BERT-base fine-tuned on a corpus of ∼400,000 organic chemistry papers and also out-performs BERT-  2 and Table 3 were performed by us.Note that we do not test BiLSTM-based architectures 22 as past work has shown that BERT-based architectures typically outperform BiLSTM-based ones. 14,16,21The performance on MaterialsBERT for each entity type in our ontology is described in Supplementary Information Section S3.

Quantifying the extracted data
Using our pipeline, we extracted ∼ 300, 000 material property records from ∼ 130, 000 abstracts.Out of our corpus of 2.4 million articles, ∼ 650, 000 abstracts are polymer relevant and around ∼ 130, 000 out of those contain material property data.This extraction process took 60 hours using a single Quadro 16 GB GPU.To place this number in context, PoLyInfo a comparable database of polymer property records that is publicly available has 492,645 property records as of this writing. 25This database was manually curated by domain experts over many years while the material property records we have extracted using automated methods took 2.5 days using only abstracts and is yet of comparable size.The composition of these material property records is summarized in

General property class
We now analyze the properties extracted class-by-class in order to study their qualitative trend.Figure 3 shows property data extracted for the five most common polymer classes in our corpus (columns) and four most commonly reported properties (rows).Polymer classes are groups of polymers that share certain chemical attributes such as functional groups.
These properties fall under the general property class as described in Table 4.The data is extracted when a polymer of that polymer class is part of the formulation for which

898
-a property is reported and does not necessarily correspond to homopolymers but instead could correspond to blends or composites.The polymer class is "inferred" through the POLYMER CLASS entity type in our ontology and hence must be mentioned explicitly for the materials record to be part of this plot.Several key trends are captured in this plot.
From the glass transition temperature (T g ) row, we observe that polyamides and polyimides typically have higher T g than other polymer classes.Molecular weights unlike the other properties reported are not intrinsic material properties but are determined by processing parameters.The reported molecular weights are far more frequent at lower molecular weights than at higher molecular weights; mimicking a power-law distribution rather than Gaussian distribution.This is consistent with longer chains being more difficult to synthesize than Figure 3: Material property data extracted from abstracts for material systems that contain a polymer from the polymer classes of polyurethane, polyimide, polyamide, polyester, and polysiloxane in each corresponding column.These are the most commonly reported polymer classes and the properties reported are the most commonly reported properties.
shorter chains.For electrical conductivity, we find that polyimides have much lower reported values which is consistent with them being widely used as electrical insulators.Also note that polyimides have higher tensile strengths as compared to other polymer classes, which is a well-known property of polyimides. 29gure 4 shows mechanical properties measured for films which demonstrates the tradeoff between elongation at break and tensile strength that is well known for material systems (often called the strength-ductility trade-off dilemma).Materials with high tensile strength tend to have a low elongation at break and conversely, materials with high elongation at break tend to have low tensile strength. 30This known fact about the physics of material systems emerges from an amalgamation of points independently gathered from different papers.In the next section we take a closer look at pairs of properties for various devices that reveal similarly interesting trends.

Knowledge extraction
Next, we consider a few device applications and co-relations between the most important properties reported for these applications to demonstrate that non-trivial insights can be obtained by analyzing this data.We consider three device classes namely polymer solar cells, fuel cells, and supercapacitors and show that their known physics is being reproduced by NLP extracted data.We find documents specific to these applications by looking for relevant keywords in the abstract such as 'polymer solar cell' or 'fuel cell'.The total number of data points for key figures of merit for each of these applications is given in Table 4. data extracted manually as reported in Ref. 31.Each data point in Figure 5a)-c) is taken from a particular paper and corresponds to a single material system.It is clear from Figure 5(c) that the peak power conversion efficiencies reported are around 16.71 % which is close to the maximum known values reported in the literature 33 as of this writing.The open-circuit voltages (OCV) appear to be Gaussian distributed at around 0.85 V. Figure 5(a) shows a linear trend between short circuit current and power conversion efficiency.It is clear that the trends observed in Figure 5a)-c) for NLP extracted data are quite similar to the trends observed from manually curated data in Figure 5d)-f).

Fuel Cells
Fuel cells are devices that convert a stream of fuel such as methanol or hydrogen and oxygen to electricity.Water is one of the primary by-products of this conversion making this a clean source of energy.A polymer membrane is typically used as a separating membrane between the anode and cathodes in fuel cells. 34Improving the proton conductivity and thermal stability of this membrane to produce fuel cells with higher power density is an active area of research.Figure 6(a) and 6(b) show plots for fuel cells comparing pairs of key performance metrics.The points on the power density versus current density plot (Figure 6(a)) lie along the line with a slope of 0.4 V which is the typical operating voltage of a fuel cell under maximum current densities. 35Each point in this plot corresponds to a fuel cell system extracted from the literature that typically reports variations in the polymer membrane.

Trends across time
We show that known trends across time in polymer literature are also being reproduced in our extracted data.A Ragone plot illustrates the trade-off between energy and power density for devices and supercapacitors are a class of devices that have high power density but low energy density.Figure 6(c) illustrates the trade-off between gravimetric energy density and gravimetric power density for supercapacitors and is effectively an up-to-date version of the Ragone plot for supercapacitors. 37Historically, in most Ragone plots, the energy density of supercapacitors ranges from 1 to 10 Wh/kg. 380][41] As seen in Figure 6(c), the majority of points beyond an energy density of 10 Wh/kg are from the previous two years, i.e., 2020 and 2021.4. Observe that the number of datapoints of the general category have grown exponentially at a rate of 6% per year.Out of the three applications considered in Figure 6(f), polymer solar cells have historically had the largest number of papers as well as datapoints although that appears to be declining over the past few years.
Observe that there is a decline in the number of datapoints as well as the number of papers in 2020 and 2021.This is likely attributable to the COVID-19 pandemic 43 which appears to have lead to a drop in the number of experimental papers published that form the input to our pipeline. 44

Discussion
A natural language processing pipeline that extracts material property records from abstracts has been built and demonstrated.This however has some limitations in practice that we describe below: 1. Materials property information is multi-modal and can be found in the text, tables, and figures in the body of the paper.Co-referencing material entity mentions across large spans of text and across figures and tables is a challenging problem.In addition to this, relation extraction of material entities and property value pairs occurring across sentences, are challenges that need to be addressed when extending this work from abstracts to full-text.
2. The current ontology used consists of the most important entity types found in materials science literature.This makes it easier to combine material and property information using heuristic rules but misses other information about the material property record such as measurement methods or measurement conditions which in most cases would influence the property value.
3. Converting polymer names to a structure (typically a SMILES string 45 ) is also a bottleneck to training downstream models as this must be done manually.Tools that can reliably and robustly convert images of chemical structures found in the literature to SMILES string are an area of future work for the community.The SMILES string so generated can be used to generate a structural fingerprint vector of the polymer which in turn can serve as the input to a machine learning model.Expanding the scope of this pipeline to images in the body of the paper would allow training downstream property models without any additional curation for converting images to SMILES strings.
Training robust property predictors in this manner would in turn allow the continuous and semi-automatic design of new materials, thus addressing a missing link in materials informatics.An example of manually converting polymer names to SMILES strings followed by the training of a property prediction model for glass transition temperature is shown in Supplementary Information Section S5.
The automated extraction of material property records enables researchers to search through literature with greater granularity and find materials systems in the range of interest.It also enables insights to be inferred by analyzing large amounts of literature that would not otherwise be possible.As shown in the section "Knowledge extraction", a diverse range of applications were analyzed using this workflow to reveal non-trivial albeit known insights.This is the first work to build a general purpose capability to extract material property records from published literature.∼ 300, 000 material property records were extracted from ∼ 130, 000 polymer abstracts using this capability.Through our web interface (https://polymerscholar.org) the community can conveniently locate material property data published in abstracts.As part of this work, we also train and release MaterialsBERT, a language model that is fine-tuned on 2.4 million materials science abstracts using Pub-MedBERT as the starting point and obtains the best F1 score across three of five materials science NER data sets tested.
Growing the extracted material property data set further would require extending this capability to the body of the paper.This would require more robust methods to associate the entities extracted using named entity recognition.A few steps also remain in order to utilize the extracted data to produce trained machine learning property prediction models.
The biggest bottleneck in the case of organic materials is obtaining SMILES strings for material entities which can then be used to generate structural fingerprints for downstream machine learning models.There is also a wealth of additional information such as processing conditions or measurement conditions that are not captured in our ontology.Addressing these bottlenecks would enable automatic and continuous updates of materials databases that can seamlessly power downstream property predictor machine learning models.

Corpus of papers
We have created a corpus of ∼ 2.4 million journal articles from the materials science domain.
The papers were downloaded from the APIs and websites from publishers such as Elsevier, Wiley, Royal Society of Chemistry, American Chemical Society, Springer Nature, Taylor & Francis, and the American Institute of Physics.The corpus used in this work is an expanded version of the corpus described previously in Ref. 12. 750 abstracts of this corpus were annotated and used to train an NER model.Furthermore, the trained NER model along with heuristic rules is used to extract material property records from the abstracts of the full corpus.

Preprocessing of documents
Because the documents in our corpus are HTML formatted, we stripped all HTML tags to parse the plain text.Moreover, we replaced HTML superscripts and subscripts (<sup> and <sub>) with plain text using the LaTeX convention of ˆ{} and {}, respectively.This is important in order to extract units of quantities as well as property values reported in scientific notation.Property values recorded in this notation were converted back to floatingpoint numbers downstream when the numeric value was to be recovered.We also mapped characters such as spaces or special characters that have multiple Unicode representations but have a similar appearance by creating a custom mapping.For tokenization, i.e., breaking up text into units known as tokens which are used for downstream processing, we used wordpiece tokenization which is the standard tokenization scheme used with BERT and BERT-based models. 1,46For instance 'The molecular weight of the result ##ant P ##LL ##A -rich polymer was enhanced .' is what a sentence would look like post-tokenization.
The word 'resultant' and the polymer 'PLLA' have been broken into sub-words tokens.This is necessary in order to tokenize arbitrary text inputs using a fixed-sized vocabulary as a larger vocabulary would increase the size of the model.Starting with a set of characters (alphabets, numbers etc), certain combinations of characters are iteratively merged and added to the vocabulary till the vocabulary reaches a certain fixed size. 47The characters to be merged are selected based on combinations that maximize the likelihood of the input text.
This typically breaks up words into meaningful subunits such as 'resultant' being separated into 'result' and '##ant' which reduces the size of the vocabulary.This does not always happen though as seen with the example of 'PLLA'.

Training MaterialsBERT
BERT-base, the original BERT model, was trained using an unlabeled corpus that included English Wikipedia and the Books Corpus. 48The training objectives included using the masked language modeling task, which masks a random subset of the input text and asks the language model to predict it and the next sentence prediction task, which determines for a given sentence pair whether one sentence follows the other in the training data. 1 The vocabulary of the tokenizer was fixed at 30,000 tokens.It is known that a domain-specific BERT encoder improves performance on downstream NLP tasks for that domain because the vocabulary used for tokenization is more representative of the application of interest and because the unlabeled text is also closer to the domain of interest resulting in "better" contextual embeddings. 14BERT-base was pre-trained from scratch using a general English language corpus. 15 Even though computationally expensive, pre-training NLP models from scratch has the advantage of creating a model with a vocabulary that is customized for the domain of interest.
To give an idea of how resource-intensive this can be, note that RoBERTa, a similarly pre-trained encoder used the computing power of 1024 V100 GPUs for one day. 49As this is not a viable route for us, we fine-tuned a model starting from previous checkpoints.The vocabulary used while fine-tuning a model in contrast remains the same as the underlying model which is a compromise we must accept.We used PubMedBERT as our starting point and fine-tuned it using 2.4 million materials science abstracts. 14The PubMedBERT model used here is itself pre-trained from scratch using the PubMed corpus (14 million abstracts from PubMed as well as full text articles from PubMedCentral), using the BERT-base architecture.We picked PubMedBERT as our starting point as its vocabulary is specific to the biomedical domain which overlaps with materials science as material entities are frequently mentioned in biomedical papers.During fine-tuning, the model weights of the checkpoint to be used are loaded and training is continued using the same training objectives as the model that is fine-tuned but using the unlabeled text from the fine-tuning corpus as the input.The hyperparameters used during fine-tuning were identical to those used to train PubMedBERT.We used the "Transformers" library for fine-tuning PubMedBERT. 50A similar strategy is employed in ChemBERT, 16 ClinicalBert, 51 and FinBERT. 52We fine-tuned PubMedBERT for 3 epochs which took 90 hours on four RTX6000 16 GB GPU's to obtain MaterialsBERT.labels on each polymer relevant document obtained from the previous step.

Abstract filtering:
As not all polymer abstracts contain material property information, the output of the NER is used as a heuristic to filter out those that do.Only abstracts with specific material entities, i.e., POLYMER, POLYMER FAMILY and MONOMER as well as the PROPERTY NAME and PROPERTY VALUE tags are allowed through this stage.This acts as a second filter to locate polymer relevant documents.

Entity extraction:
The material entities, (PROPERTY NAME, PROPERTY VALUE) and MATERIAL AMOUNT entities are extracted and processed separately.
6. Co-reference material entities: This step is applied to co-reference all mentions of the same material entity.A common example of this is when a material is mentioned next to its abbreviation.We used the abbreviation detection system in Chem-DataExtractor 3 to find material entity abbreviation pairs.In addition, we co-reference material entities that are within a Levenshtein distance 53 of one.Co-referencing is a tractable problem in abstracts compared to full-text papers as there are no long-range dependencies and typically no anaphora resolution is required. 54 Normalizing polymer names: Polymers can have several different variations in names referring to the same chemical entity.In this step, we normalize these variations to the most commonly occurring name for that particular polymer.For instance, 'poly(ethylene)' and 'poly-ethylene' occurring in different abstracts are both normalized to 'polyethylene'.This is done using a dictionary lookup on a dataset of polymer name clusters that were normalized using the workflow described in Ref. 26.Note that we do not normalize all polymer names but only the ones which are included in our dictionary.
In practice, this includes most commonly occurring polymers that have multiple names in the literature.

Extract Property Value pairs:
The PROPERTY NAME and PROPERTY VALUE tag are associated by co-occurrence within a context window.The numeric value of the property is separated from the units using regular expressions and all parsed property values are converted to a standard set of units.The unit used is the most commonly reported unit for that particular property.Any errors reported with the numeric value are also parsed using regular expressions.9. Extract Material amounts: Entities with the MATERIAL AMOUNT tag are extracted and the closest material entity within a context window is associated with it.
10. Relation extraction: In order to obtain a material property record, it is necessary to associate the material entities and the property value pair that correspond to a single record.This problem has been addressed in the literature using supervised methods. 55,56However, the annotation process for relation labeling is time-intensive and hence we employ heuristics in this work to obtain relations between entities.To associate material entities with property value pairs, we associate the closest material entity tagged in the same sentence as the property value pair.If no such material entity is found then all the material entities mentioned in the abstract are associated with the property value pair.This is because, most commonly, an abstract mentions a major material system reported in the paper and reports its measured property values.
This step is reasonable in abstracts, which report this information compactly, using heuristic rules.In contrast, the body of the paper would require coreferencing with information found in tables and figures to extract material property records. (

S3. Performance of MaterialsBERT on annotated polymer abstracts NER dataset
The performance of the various entity types for MaterialsBERT, the best-performing model along with the distribution of various entity types in the annotated dataset of polymer abstracts is shown in Table S1.It is clear that there is a positive correlation between the number of occurrences of each entity type and the performance of the corresponding entity type.The ORGANIC entity type has a low F1 score likely because it is difficult to distinguish between MONOMER and ORGANIC entity types.The POLYMER FAMILY entity type has similarities with the POLYMER entity type which likely lowers the F1 score for this entity type as well.

S4. Details of extracted data
The number of entities extracted (accounting for variations in case) from the ∼ 300, 000 material property records is shown in Table S2.Note that these material entities are not unique materials as for instance, 'silica' and 'SiO 2 ' would be counted as separate entities but are the same material.Similarly for property names, 'T g ' and 'glass transition temperature' are counted as separate entities, even though they correspond to the same property.The histogram of number of records extracted for each property is shown in Figure S1.
Only property entities with at least 10 material property records are shown in this plot.
Observe that the number of material property records appears to be power law distributed with a small number of properties having a large number of records associated with them  S3 while the majority of property entities only have a small number of associated records.
We threshold this plot at 500 material property records and report the number of records associated with all properties above this threshold in Table S3.Note that common variations in names for each property such as abbreviations (e.g.IEC for ion exchange capacity) or plural forms (e.g.activation energies) were accounted for in order to compute the number of records.These account for a total of 111279 records, i.e., 37 % of ∼ 300, 000 material property records.Note that device properties such as power conversion efficiency, open circuit voltage etc listed in Table 5 of the main paper are listed in Table S3 as well but are more in number as they are not restricted to any particular application.Observe that the test RMSE in both cases is similar

Index
In addition to the qualitative trends shown in the main paper that demonstrate the utility of our workflow, we also demonstrate how the data collected through our workflow can be used as inputs to machine learning models.We trained gaussian process models with a Matern kernel 57 to predict T g using the material property data extracted from our workflow.SMILES strings 45 were used to encode the structure of the polymer.The SMILES strings for a randomly selected subset of NLP extracted neat polymers T g records were added manually.The SMILES string was used as the input to fingerprint the polymer and the fingerprint vector along with the literature extracted T g value was the input to the machine learning model.The polymer SMILES string is converted to a feature vector using structural descriptors described elsewhere. 28The feature vector consists of atomic triples,

Figure 1 :
Figure 1: Pipeline used for extracting material property records from a corpus of abstracts

Figure 2 :
Figure 2: Model architecture used for named entity recognition: Each token in the input sequence is converted to a contextual embedding by a pre-trained transformer model which is then input to a single layer neural network.The output of the neural network is the entity type of the input token.

base 1
across the NER data sets tested.BioBERT 15 was trained by fine-tuning BERT-base using the PubMed corpus and thus has the same vocabulary as BERT-base in contrast to PubMedBERT which has a vocabulary specific to the biomedical domain.Ref. 21 describes the model MatBERT which was pre-trained from scratch using a corpus of 2 million materials science articles.Despite MatBERT being a model that was pre-trained from scratch, MaterialsBERT outperforms MatBERT on three out of five datasets.While the vocabulary of MatBERT and MaterialsBERT are both relevant to the domain of materials science, this performance difference can likely be attributed to the fact that PubMedBERT, the initial model for MaterialsBERT was pre-trained on a much larger corpus of text (14 million abstracts and full text).All experiments shown in Table

Figure 4 :
Figure 4: Tensile Strength Vs Elongation at break for films demonstrating the strengthductility trade-off

Figure 5 :
Figure 5: Co-relations between key properties extracted automatically from literature for polymer solar cells a) Power Conversion Efficiency against short circuit current b) Power Conversion Efficiency against fill factor c) Power Conversion Efficiency against open circuit voltage.Co-relations between key properties extracted manually from literature for polymer solar cells d) Power Conversion Efficiency against short circuit current e) Power Conversion Efficiency against fill factor f) Power Conversion Efficiency against open circuit voltage.Observe that the trends in this figure match well with NLP extracted data in Figure 5. Reproduced here with permission from Ref. 31

Figure 6 (Figure 6 :
Figure 6(b) illustrates yet another use-case of this capability, i.e., to find material systems lying in a desirable range of property values for the more specific case of direct methanol fuel cells.For such fuel cell membranes, low methanol permeability is desirable in order to prevent the methanol fuel from crossing the membrane and poisoning the cathode side.36High proton conductivity is simultaneously desirable.The box shown in the figure illustrates the desirable region and can thus be used to easily locate promising materials systems.

Figure 6 (
Figure 6(d) and Figure6(e) shows the evolution of power conversion efficiency of polymer solar cells for fullerene acceptors and non-fullerene acceptors.These are the two major classes of acceptors which along with a polymer donor form the active layer of a bulk heterojunction polymer solar cell.Observe that more papers with fullerene acceptors are found in earlier years with the number dropping in recent years while non-fullerene acceptor based papers have become more numerous with time.They also exhibit higher power conversion efficiencies than their fullerene counterparts in recent years.This is a known trend within the domain of polymer solar cells reported in Ref.42.It is worth noting that the authors realized this trend by studying the NLP extracted data and then looking for references to corroborate this observation.

Figure 6 (
Figure 6(f) shows the number of datapoints extracted by our pipeline over time for the various categories described in Table4.Observe that the number of datapoints of the general

Figure S1 :
Figure S1: Histogram of number of material property records for each property entity.The threshold represents the cut-off above which the number of material property records is reported for each property in TableS3

Figure S2 :
Figure S2: Glass Transition Temperature data a) Parity plot for machine learning model trained using literature extracted glass transition temperature data, b) Parity plot for machine learning model trained using curated glass transition temperature data with the same number of points as used in a).The test set in both cases is drawn from curated data.Observe that the test RMSE in both cases is similar

Table 1 :
Description of each entity type in the ontology used for annotating polymer abstracts

Table 2 :
Performance of various pre-trained BERT-based encoders on the test set of polymer abstracts.Values are reported in %

Table 3 :
Performance of various BERT based encoders on the test sets of publicly available materials science NER datasets.Values are reported in %

Table 4
not of the POLYMER or POLYMER CLASS entity type.To compute the number of unique neat polymer records, we first counted all unique normalized polymer names from records that had a normalized polymer name.This accounts for the majority of polymers with multiple reported names as detailed in Ref.26.Out of the remaining neat polymer records that did not have a normalized polymer name, we then counted all unique polymer names (accounting for case variations) and added them to the number of unique normalized polymer names to arrive at the estimated number of unique polymers.The number of extracted data points reported in Table4is higher than the figures shown later for specific properties (grouped into a few property classes) that are utilized later on.For the general property class, we compute the number of neat polymers as the material property records corresponding to a single material of the POLYMER entity type.Blends correspond to material property records with multiple POLYMER entities while composites contain at least one material entity that is as additional constraints are imposed in each case in order to better study this data.For the general property class, we note that elongation at break data for an estimated 413 unique neat polymers was extracted.In contrast, Ref. 27 used 77 polymers to train a machine learning model.For tensile strength, an estimated 926 unique neat polymer data points are extracted while Ref. 28 used 672 data points to train a machine learning model.Thus the amount of data extracted in the aforementioned cases is already comparable to or greater than the amount of data being utilized to train property predictors in the literature.Note that Table4accounts for only 39207 points which is 13 % of the total extracted material property records.More details on the extracted material property records can be found in Supplementary Information Section S4.The reader is also encouraged to explore this data further through https://polymerscholar.org

Table 4 :
Number of material property records extracted for several key polymer properties and figures of merit for certain applications 7) Court, C. J.; Jain, A.; Cole, J. M. Inverse Design of Materials That Exhibit the Magnetocaloric Effect by Text-Mining of the Scientific Literature and Generative Deep Learning.Chemistry of Materials 2021, 33, 7217-7231.Inorganic Synthesis recipes: 24 This is a dataset of 230 inorganic synthesis paragraphs annotated using 21 entity types that are relevant in the context of materials synthesis.15 paragraphs each are used for validation and testing and the remaining 6. MATERIAL AMT: Numerical quantity + unit denoting the amount of a material like additive or blend for which a property value is reported.This is typically wt % or mol.7.PROPERTY NAME: Name of property measured.Label properties for which a corresponding numeric value is reported in the abstract.Label the property name and abbreviation in bracket (if present) separately.8.PROPERTY VALUE: Numeric value + unit for a reported PROPERTY NAME.4.ChemRxnExtractor: 16 This is a dataset of 329 organic synthesis paragraphs in which the product of the synthesis is labeled using the BIO scheme.The dataset is split into 251, 41, and 37 paragraphs in the train, validation, and test set respectively.

Table S1 :
Performance of an NER model using MaterialsBERT as the encoder across various entity types in the ontology used in this work on the test set of polymer abstracts.Values are reported in %

Table S2 :
Number of entities extracted from the corpus of polymer relevant abstracts for key entity types from the ontology used in this work