Main

Biodiversity loss is now widely recognized as a major threat to planetary health1,2,3. Halting the loss requires that the basic building blocks of biodiversity are known, so that changes can be recorded, drivers of change can be identified and appropriate policy actions can be implemented. However, much of the terrestrial animal diversity belongs to hyperdiverse invertebrate clades that are so poorly known4,5 that it is difficult to obtain this critical information. For example, only 0.17 G of the 2.16 G records in the Global Biodiversity Information Facility pertain to arthropods. By comparison, 67% of Global Biodiversity Information Facility records relate to birds, although birds account for only 10,000–20,000 species (0.2%) of the estimated 8–10 million multicellular species worldwide6,7. These numbers alone reveal the size of the knowledge gap for many truly diverse clades that due to their current position in the information shadow have been called ‘dark taxa’8.

To allocate resources for discovering and conserving species, it is crucial to establish the relative contribution of different taxa to overall biodiversity. Only in this way can the most diverse and abundant taxa be given adequate attention. Identifying these taxa is furthermore important for understanding the basic structure of the living world, and for gaining insights into how community composition is shaped by evolutionary, biogeographic or ecological factors9. Where such analyses have been carried out—for example, for plants and snakes10—they have revealed that a few clades dominate communities across the world11. Unfortunately, corresponding information is lacking for arthropods. This is a striking shortcoming, given that arthropods are found worldwide, functionally important12 and currently undergoing major declines in diversity and abundance13,14.

In this Article, we analyse the taxonomic patterns among flying insects sampled by Malaise traps in different habitats, climates and biogeographic regions. Malaise traps are widely used in global biomonitoring programmes because they provide standardized and efficient tools for collecting diverse communities of flying insects and semi-aquatic taxa15,16,17. Similar to all other trap types, they only subsample the insect communities. For example, Malaise traps rely on the passive interception of insect flight paths, and collect those insects that climb towards the highest point of the trap (Supplementary Fig. 1). For this reason, strong and active fliers like dragonflies (which largely avoid the traps) or beetles (which tend to drop to the ground when encountering an obstacle) are under-represented. However, overall, Malaise traps are so effective at sampling flying insects that sample processing is a major challenge due to high specimen and species yields15,18. In addition, most specimens caught in Malaise traps cannot be identified, because many species are undescribed and relevant taxonomic expertise is either non-existent or dwindling6. Fortunately, recent advances in large-scale DNA barcoding with new sequencing technologies allow for processing large numbers of specimens rapidly and cost-effectively19,20. Using molecular species delimitation methods, these data can then be converted into estimates of species diversity without formal description of the component taxa and most species can be assigned to major insect clades for analysis of community structure.

We here determine the taxonomic composition of Malaise trap samples21 from five biogeographic regions, eight countries and diverse habitats. In total, our material encompasses >225,000 specimens belonging to >25,000 species living in habitats ranging from temperate meadows to tropical rainforests. We discover surprising congruence with regard to which 20 insect families are dominant components of flying arthropod communities worldwide (accounting for >50% of species and specimens in each sample). When we compare family-specific diversity with taxonomic attention, we find that most of the particularly diverse and abundant taxa are poorly known and suffer from persistent taxonomic neglect. In other words, a very large proportion of terrestrial animal biodiversity is not only unknown to science, but will also remain so for the foreseeable future unless such ‘dark taxa’ become a preferred target for biodiversity science.

Results

Our study comprises 225,261 barcoded arthropods belonging to 458 families. They represent the insect diversity obtained from 39 traps across eight different sites and all continents excluding Australia and Antarctica. Applying a species-delimitation threshold of 3% sequence similarity20,22 reveals that each Malaise trap yielded anywhere from 69 to 3,426 molecular operational taxonomic units (mOTUs). When these mOTUs were assigned to higher arthropod clades, we found that, on average, 57.2% and 19.0% of the species in a trap belonged to the orders Diptera and Hymenoptera (Supplementary Fig. 3.1). When examined at the family level, 61.7% of the specimens and 51.9% of the species in each trap belong to a set of only ten insect families (henceforth referred to as the ‘top 10 families’). The next ten families added only 9.7% and 12.2% of specimens and species, respectively (Fig. 1: see the ‘top 20 families’). Nearly one-fifth of the species per site (average 20%) belonged to a single dipteran family, Cecidomyiidae (Fig. 2a).

Fig. 1: Congruence in the relative contribution of insect families to specimen abundance and species richness.
figure 1

a,b, Each chart shows the taxonomic composition of a sample obtained by an individual Malaise trap at a specific site. The inner circle represents the proportion of biodiversity in the top 10 (green), next 10 (blue) and remaining families (grey). The outer ring shows what proportion of biodiversity belongs to the top 10, next 10 and the remaining families. Black is used to illustrate the extraordinary diversity of Cecidomyiidae (Diptera). All charts are scaled relative to number of specimens (a) and species (b) at each site. Map made with Natural Earth. Supplementary Fig. 2 provides precise geolocations for each site.

Full size image
Fig. 2: Proportional species richness of top 20 insect families in Malaise traps.
figure 2

a,b, Consistency in community composition of insects as shown by proportional (%) species richness of the top 20 families among sites (x-axis log-transformed) (a) and proportion of variance absorbed by the first axis in a PCA of variation in the proportion of species richness per insect family (b). Individual violins in the violin plot in a show the average proportional contribution ± standard deviation for each individual family.

Full size image

The relative species richness of individual insect families showed remarkably similar patterns across the globe, with the top 20 families (Fig. 1b) accounting for 41.2–72.3% of the total species richness regardless of continent or climate (Canada: 63.9%, Egypt: 47.3%, Germany: 65.8%, Honduras: 65.5%, South Africa: 54.5%, Pakistan: 49.4%, Saudi Arabia: 41.2% and Singapore: 72.3%). Yet, the high species richness of these families was not correlated with clade age (r = 0.0039; n = 10,023, P = 0.70, Supplementary Fig. 5).

The dominance of specific insect families across sampling sites is perhaps best illustrated by the consistent differences in species richness among families. In our main dataset (39 Malaise traps), 66.4% of the variation among traps in log-transformed species richness was explained by family. These results persisted irrespective of the species delimitation method used (with an adjusted R2 of 66.4% when species were delimited by objective clustering23 at a 3% distance threshold, and an adjusted R2 of 64.9% when species were delimited by Assemble Species by Automatic Partitioning (ASAP)24). In our expanded dataset (Methods), which retained sample-level resolution for traps placed in Germany and Canada (56 datasets), the corresponding figure was 67.1% for objective clustering at 3%. The only qualitative difference in results between the main and expanded datasets was whether Mycetophilidae (Diptera) and Crambidae (Lepidoptera) were included among the top 20 families. Note that this list of top 20 families is furthermore robust to changes in family designations. This was tested by merging clades with their sister clades based on recent phylogenies to thereby account for taxonomists’ disagreements on rank (Supplementary Table 3). Nineteen of the top 20 families remain in the list, and the only change involved the replacement of one lepidopteran clade (Gelechiidae + Cosmopterigidae replaced Crambidae). Unsurprisingly, the convergence of taxonomic composition was even stronger at the order level (90.6% of log-transformed species richness based on adjusted R2 and objective clustering at 3.0% distance threshold, Supplementary Fig. 3.1).

The convergence of relative species richness was also evident from a principal component analysis (PCA), which revealed that >60% of the variance can be explained by a single principal component (Fig. 2b and Supplementary Figs. 3.2b and 3.3b). In contrast, analysis of species turnover between the major regions showed that almost all species in the top 20 families (97.6%) were found at a single site only (Supplementary Table 2). In other words, variation in community composition was largely attributable to variation in the relative contribution of distinct species from a small set of families.

Given the disproportionate contribution of a few families to insect diversity across the world, we next examined whether the globally dominant taxa have attracted appropriate taxonomic attention. To characterize taxonomic attention or neglect, we first defined a ‘neglect index’ (NI) as the ratio between the number of mOTUs found across the Malaise traps for a given family and the total number of species described as listed in the Catalog of Life (CoL: https://www.catalogueoflife.org/). An NI value of 1 signals that we detected as many mOTUs in the current set of 39 Malaise traps as have been formally described for the entire world, whereas a low NI value reveals that we found only a tiny proportion of all species described so far. We then investigated how this index is correlated with species richness and body size. We found a positive correlation between the log NI and the log number of mOTUs detected in our samples (main dataset: r = 0.61, n = 20, P = 0.004; expanded dataset: r = 0.54, n = 20, P = 0.01) (Fig. 3 and Supplementary Fig. 4). Moreover, we saw a strong negative relationship between the NI and the number of species described per decade between 1980 and 2019 (main dataset, r = 0.44, n = 80, P = 0.00003; expanded dataset, r = 0.48, n = 80, P = 7.6 × 10−6). In other words, the more neglected a taxon is, the fewer new species are described per decade.

Fig. 3: Taxonomic neglect, species diversity and taxonomic activity dedicated to the top 20 families.
figure 3

a,b, Taxonomic neglect (as expressed by the NI) increases with the species diversity of the target taxon (a), and taxonomic neglect decreases with increasing body size of the target taxon (b). c, The more neglected a taxon is, the less taxonomic attention is dedicated to it (with no sign of improvement over time). d, Likewise, the number of authors publishing monographic work on the top 20 families shows no increase over time. The stacked bars show the proportion of families with 0 (white), 1 (brown), 2 (blue), 3 (yellow), 4 (grey), 5–9 (green) and 10+ (orange) authors having described ≥50 species in a decade (see also Supplementary Fig. 4).

Full size image

Furthermore, we see no signs of any increase in taxonomic attention paid to neglected taxa over time: the slope of the relationship between species richness and taxonomic neglect showed no detectable change over time (non-significant interaction decade × NI for the main dataset; analysis of variance (ANOVA): F6,72 = 0.86, P = 0.53, expanded dataset, ANOVA; F6,72 = 0.80, P = 0.58). In a similar vein, the number of taxonomists involved in monographic work (that is, the number of taxonomists describing ≥50 species in a decade) shows no increase over time (Fig. 3c). In fact, the number of such taxonomists involved in monographic revisions targeting the top 20 families was the lowest for the decade of 2010–2019 (Fig. 3c: see the white part of the columns).

In terms of the drivers of taxonomic neglect, the natural logarithm of NI increased significantly with species diversity (the log number of mOTUs detected in our samples; coefficient ± standard error (SE): 0.656 ± 0.152, t = 4.318, P = 0.0005) and decreased with the logarithmic mean body size of the taxon (coefficient ± SE: −1.102 ± 0.188, t = −5.877, P = 0.00002). For the expanded dataset, the corresponding numbers for species diversity were coefficient ± SE: 0.721 ± 0.156, t = 4.631, P = 0.0002 and for body size coefficient ± SE: −1.222 ± 0.203, t = −6.029, P = 0.00001.

Discussion

With more than 80% of species undescribed, insects arguably remain the key taxonomic challenge for understanding animal diversity. We here reveal that the same 10–20 clades ranked as families dominate communities of flying insects around the world, when sampled by Malaise traps. This convergence is remarkable, given that the samples were collected across several distinct climatic zones and habitat types, including tropical rainforests, montane forests, cedar savannah, bushwillow woodlands, thorn veld, mangroves and marshes. The biodiversity challenge posed by these ‘dark taxa’ is formidable given their high species diversity and turnover at most sites. A prime example is Cecidomyiidae (gall midges), a globally hyperdiverse taxon that dominates insect communities in terms of species counts and abundance25. Yet, regardless of its widespread dominance, this family has received little taxonomic attention.

In striking evidence for the consistency of community composition, we find that two-thirds of the variance across Malaise trap samples is explained by family membership of a species. This raises the question of how such a pattern can emerge across widely different ecosystems and large geographic scales. One explanation could have been that dominant insect families were older, and thus had spread earlier and diverged for a longer time in each part of the world. However, we detected no correlation between species richness per family and clade age (r = 0.0039; n = 10,023, P = 0.70, Supplementary Fig. 5). Another potential explanation for large-scale convergence in community composition might have been widespread species appearing in communities around the world. However, fewer than 3% of species in the top 20 families were found at multiple sites (1–9% in any given family, Supplementary Table 2). Instead, the convergence of taxonomic composition is more likely due to high diversification rates and/or high evolutionary plasticity of taxa ranked above the species and below the family level. Such pronounced adaptability may have allowed these taxa to diversify across habitats and climatic zones. In support of this notion, most species belonging to the top 20 families rely on resources widely available in most habitats (for example, plants, fungi and insect hosts) that are likely to require species-specific adaptations for exploitation26,27. This hypothesis should be tested by clade-specific research given that the species diversity within subclades ranked below the family level often varies considerably. For instance, of the ten subfamilies of Cecidomyiidae, one (‘Cecidomyiinae’) contains >70% of described gall midge diversity. Similarly, a single subfamily (Psychodinae) among the seven subfamilies of Psychodidae contains 59% of described drain fly diversity (CoL: https://www.catalogueoflife.org). Similarly, 45% of described diversity of ants (Formicidae) derives from one of its 20 subfamilies (Myrmicinae). The next step in understanding these patterns would be detailed analyses of diversification rates and colonization events for clades below the family and above the species level for the dominant taxa. Such analysis applied to the much less diverse clade comprising snake species identified a few rapidly diversifying lineages, which dominate globally11. These clades (for example, Colubrinae) are apparently superior colonizers and ecologically successful in geographically disparate regions regardless of the presence of other taxa. However, compared with most insect clades encountered in our study (for which the mean age is 158 mya), colubrines are very young (with an age of <50 mya), and the patterns here reported for insect have thus evolved over a much greater span of time.

What should be noted is that our study analyses only those insect taxa that are captured in Malaise traps. These traps are particularly effective for sampling flying insects17,28 and used widely in large-scale insect biomonitoring programmes15,16 even though they are known to mostly target weak fliers active at ground level. The samples thus contain few canopy species, strong fliers and taxa that drop to the ground when encountering an obstacle (for example, many beetles). Such taxa are best sampled with targeted sweep-netting or other traps such as pitfall, leaf-litter, flight intercept, suction or automated light traps29,30. It will thus be critical to repeat similar comparative and large-scale analyses of the taxonomic composition of insect communities based on samples obtained with these methods. Such analyses will eventually reveal whether the order-level dominance of Diptera and Hymenoptera31 in Malaise trap samples is widespread enough that these orders will surpass the global species richness of Coleoptera and Lepidoptera32. In tentative support of such dominance, we note that, among the top 20 families identified in the current study, 10 are dipteran (Cecidomyiidae, Ceratopogonidae, Chironomidae, Psychodidae, Sciaridae, Phoridae, Dolichopodidae, Muscidae, Sphaeroceridae and Chloropidae) and 6 hymenopteran (Platygastridae, Formicidae, Braconidae, Ichneumonidae, Eulophidae and Bethylidae) (Fig. 2a). The diversity of the top taxa observed is exceptionally high by absolute standards, and the species turnover is very high. Thus, we would like to stress that the patterns reported for Malaise traps cannot be attributed to an over-representation of taxa although some other taxa may be under-represented.

Overall, the current study is a demonstration of the problems that can be caused by taxonomic impediments. In this case, they are used to interfere with understanding the biodiversity community patterns of flying insects. What allowed us to now address this community structure at the species and family level was a combination of new high-throughput species discovery methods, achieved via large-scale barcoding33 and efficient taxonomic assignment techniques. Such technological developments help with community analysis, but also provide partial solutions to overcoming taxonomic impediments. Large-scale barcoding can generate sequence information for large numbers of small insects at a cost of less than 10 cents per specimen in laboratories that require minimal equipment33. The process becomes even more efficient when imaging and specimen handling is robotic and/or semiautomatic and the images are used to train convolutional neural networks34. In the future, this may ease research on dark taxa by allowing for identifications based on images alone. Efficient barcoding and imaging are essential for implementing a ‘reverse-workflow taxonomy’8,19,35,36, where bulk samples are first sorted on the basis of DNA barcodes, before nuclear markers or morphology are used to test whether the clusters delimited by barcodes constitute species. If followed by automated ways to generate species descriptions, it will efficiently help with addressing the species description shortfall for dark taxa.

As biodiversity loss is threatening environmental health globally, obtaining unbiased biodiversity information across all taxa is crucial. Such unbiased information will be important for complementing the vast amount of data already compiled for large and charismatic species37. By extension, it will bring the taxa that dominate ecosystems in terms of species diversity, abundance and biomass38 into the realm of future biodiversity assessments.

Arguably, one of the most worrying findings of our study is the persistent taxonomic neglect of some of the most important insect families. We found that the more a family contributed to insect communities around the world, the more it has been neglected. Furthermore, we found no evidence for any increase in the taxonomic attention being paid to neglected taxa over time. On the contrary, the neglect proved most severe for the most diverse insect families. It is particularly serious for taxa with small average body size, a pattern that is also well known for beetles where large-bodied species are discovered and described earlier than small species39,40. Neglecting species-rich taxa containing many small species thus seems a major shortcoming of modern biodiversity science.

As one among many unfortunate outcomes, the neglect of dominant taxa compromises current estimates of global species richness. These types of estimates frequently use ratios of species richness across families (for example, ratio of butterflies diversity to known insect diversity in Britain39) as a basis for extrapolation. If the richness estimates for dark taxa were incorrect, then such estimates would be severely affected. For instance, in the United Kingdom, only 2.7% of described insect diversity belong to Cecidomyiidae41, compared with an average of 20% found in the Malaise trap communities analysed here. Assuming that the true diversity of Cecidomyiidae is closer to 20% of the British fauna, then the global species richness estimate would shift from 5.4–7.2 million to 6.5–8.7 million species—even though we are here revising the diversity estimate for only a single dark taxon (Supplemenry Material 1). Thus, the neglect of dark taxa could severely affect our perception of how life on Earth is organized, and there is an urgent need to start intensive work on these taxa to reveal the true species diversity of our planet (‘dark taxon biology’).

Overall, our study suggests that biodiversity research on 10–20 insect families should be a global priority, given the immense gap between our state of knowledge versus the likely importance of these taxa. To understand the functional importance of the key taxa uncovered, similar scalable and new approaches are also needed to reveal the biology of these species including their interactions with other species. Such progress can be achieved through new approaches to taxonomy such as the reverse workflow19. They can facilitate the collaboration between taxonomists and molecular ecologists, as the same vouchers can be used to describe species and to gain insights into their biology (for example, by sequencing gut content). Close collaboration of this type will allow for a step change in biodiversity research, conditional on adequate resources being directed to priority taxa.

Methods

Datasets, sample collection and processing

The study used DNA barcode generated for full Malaise trap samples across the globe. New datasets were generated for 24 samples from different habitat types in Singapore. Samples were recovered between 3 May 2019 and 9 May 2019 from traps placed at the site for a week before the collection date. All 24 samples were preserved in molecular-grade ethanol before processing all specimens using the high-throughput DNA barcoding pipeline described below. Six of these samples had <100 specimens and were subsequently excluded from analysis. The remaining 18 samples covered a terrestrial forest (5 traps), a mangrove forest (7 traps), coastal forests (3 traps) and a marsh (3 traps). The new data for Singapore were complemented with data from published studies that had sequenced all specimens from a large number of Malaise traps placed in the following countries: Germany16, Canada42,43, South Africa44, Pakistan45, Saudi Arabia45, Egypt45 and Honduras46 (Supplementary Table 1). To avoid strong geographic biases, we used only data for 9 of the 20 Malaise traps from South Africa (Kruger National Park), each representing a different habitat. For the study by Telfer et al. (2015) (ref. 43), we limited our analysis to the largest sample.

DNA sequencing, barcoding and identification

Insects from Malaise traps placed in Singapore were processed in a similar approach to Yeo et al. (2021) (ref. 20) and Srivathsan et al. (2021) (ref. 33) in that we used next-generation sequencing barcoding methods35. Briefly, DNA was extracted using 10–30 μl HotSHOT47 per specimen and heated to 65 °C for 18 min, followed by 98 °C for 2 min, after which an equal volume of neutralization buffer was added. A 313 bp fragment of cox1 was amplified using primers mlCO1intF and 5′-GGWACWGGWTGAACWGTWTAYCCYCC-3′ (ref. 48) and jgHCO2198: 5′-TANACYTCNGGRTGNCCRAARAAYCA-3′ (ref. 49). The primers were tagged with a 13 bp tag at the 5′ end designed for MinION-based barcoding18,33 and 9 bp tags for Illumina-based barcoding19 (Supplementary Data 1). Polymerase chain reactions (PCRs) were conducted in 96-well plates using one negative control per plate, and each PCR mix contained 8 μl Mastermix (CWBio), 0.5 μl bovine serum albumin (1 mg ml−1), 0.5 μl MgCl2 (25 mM), 1 μl each of primer (10 μM) and 4–7 μl of template DNA. The cycling conditions were: 5 min initial denaturation at 94 °C followed by 35 cycles of denaturation at 94 °C (30 s), annealing at 45 °C (1 min) and extension at 72 °C (1 min), followed by final extension of 72 °C (5 min). A subset of 8–15 products per plate were run in agarose gels to assess PCR success. Samples were pooled and purified using Ampure XP beads (Beckman Coulter). Pooled samples were sequenced either using Illumina HiSeq 2500 (2 × 250 bp) or MinION (Oxford Nanopore Technologies). Illumina sequencing was outsourced while MinION sequencing was conducted in-house using an R9.4 flowcell. Libraries were prepared using the SQK-LSK109 Ligation Sequencing Kit with two recommended modifications33. Firstly, the end-repair reaction consisted of 50 μl of DNA in molecular-grade water, 7 μl of Ultra II End-prep reaction buffer (New England Biolabs) and 3 μl of Ultra II End Prep enzyme mix (New England Biolabs). Secondly all clean-ups using Ampure beads were conducted at 1× ratio. Fast basecalling model as implemented in Guppy was used as high-accuracy basecalling was not available at the time of data processing. The 1D MinION reads that have estimated raw accuracy ~90% (ref. 50) were then converted into DNA barcodes using error corrections that have been shown to yield DNA barcodes that are virtually identical to barcodes obtained with Sanger or Illumina sequencing (99.99% accuracy18).

Data analysis of the Illumina reads started with paired-end read merging using PEAR51. Reads were demultiplexed allowing for up to a 2 bp mismatch in primer sequences, while no mismatch was allowed in the tag sequence. Demultiplexed reads for each specimen were merged to form unique sequences, and only amplicons having at least 50 sequences were processed further. A dominant sequence was identified, and if it had a read count exceeding 10, it was ‘called’ as the DNA barcode for the specimen, as long as it was also at least five times as common as the second dominant sequence. MinION sequence data were processed using minibarcoder18,52, which both demultiplexes the data and calls the barcodes. The final consolidated barcode sets were used for further analysis.

Barcodes were clustered at 1% using objective clustering (see below), and specimens were sorted physically on the basis of their cluster assignments. For Singapore samples, each cluster was morphologically identified to family. For Lepidoptera specimens, as well as for a small number of other specimens where morphological identification was not possible, we assigned specimens to families on the basis of DNA characters alone. This was done by conducting BLAST against NCBI-nt database as well as searches against the BOLD Systems Identification engine (https://boldsystems.org/index.php/IDS_OpenIdEngine). A taxonomic assignment to family was accepted if there were no conflicting family-level matches in the top 20 unique matches. For all other morphologically identified specimens, DNA-based identification was examined and any conflict with morphology was resolved through re-examination of morphology. If a conflict persisted, the specimen was not identified to family. Taxonomic classifications for published studies were based on metadata provided by the studies. These studies employed various methods of identification including morphology, matches on BOLD, and tree-based identifications. It was noted that several Hymenoptera identified on the basis of morphology as Scelionidae matched Platygastridae on BOLD Systems. This is probably due to recent classification changes. The ‘old’ Platygastridae (before 2007) was treated only as Platygastroidea (Platygastridae = Scelionidae) until very recently when it has been split into several families53. We here follow several other recent studies43,45 that have used the old circumscription of Platygastridae.

Species delimitation

Before species delimitation, we excluded sequences that contained a stop codon when translated using the invertebrate mitochondrial genetic code. Secondly, to ensure that the large datasets had sufficient sequence overlap for multiple sequence alignments, short barcodes were excluded. Any sample/trap that contained <100 barcode sequences was excluded. For datasets containing 313 bp barcode sequences, the length cut-off was 300 bp, while the cut-off was 500 bp for datasets containing 658 bp barcodes. Barcodes were aligned using MAFFT v7 (ref. 54). Species delimitation was conducted using objective clustering, which is a distance-based clustering algorithm originally described in Meier et al. (2006) (ref. 23). Species delimitations were also conducted using another distance-based approach Assemble Species by Automatic Partitioning (ASAP)24 and a tree-based approach (Poisson Tree Processes or PTP)55. For PTP-based species delimitation, phylogeny was constructed using RaxML (v8.2.5) and species delimitation was conducted using mPTP (v 0.2.4,–single,–ml). Most analyses initially used mOTUs obtained with objective clustering using a 3% distance threshold before testing the results with ASAP and PTP. We find that the results are very similar irrespective of clustering method or distance threshold (Supplementary Table 1). Species delimitations were performed for individual datasets independently. An estimate for total species diversity was obtained using USEARCH (v 11.0.667) (ref. 56) cluster_fast (-sort length -id 0.97) for the 225,261 sequences used in the study.

Statistical analyses of community composition

All analyses were conducted in R v4.1.2 (ref. 57). Analyses were limited to insects (that is, spiders, Collembola and so on were excluded: see list in Supplementary Data 2: Tables 14 and 15) and species that could be identified to family (leading to exclusion of 0.05–11.6% of the mOTUs, Supplementary Table 1). Overall sequences from 225,261 specimens were analysed. Two different datasets were studied: one where all sequences available for the same Malaise trap were combined (39 traps, main dataset) and one where the sample-level resolution for the datasets from Germany and Canada was retained (56 datasets, expanded dataset). This was feasible due to availability of high-quality metadata such that weekly samples could be treated separately. Community composition at the family level was analysed using a linear model. Here we first logarithmically transformed the proportion of mOTUs for each dataset, adding 0.01 to zero proportions (since (log(0) is undefined). We then modelled the transformed response variable as a function of family [lm(log(Proportion + 0.01)~Taxon,data = dataset)], using adjusted R2 values as the key statistic of variance explained. Furthermore, we ran a PCA on the community matrix of each site (with cell values equalling the proportion of species richness per family) using rda. We set scale as FALSE, and used a barplot of relative eigenvalues to assess the percentage variation explained by each principal component.

The top 20 families were identified on the basis of ranking of average proportion of mOTUs per family. To test whether the subjective nature of family ranks influence the results, we also examined which taxa were in the top 21–30 taxa. We then merged each with its sister clade on the basis of recent phylogenies, to test whether the merge would generate a taxon that would be included in the list of top 20 families. Next, we examined whether the high number of species in these clades across Malaise trap samples was due to high species-level dispersal rates. We analysed species turnover across ‘sites’. ‘Sites’ were broadly defined as Canada, Egypt, Germany, Honduras, Saudi Arabia, Pakistan, South Africa and Singapore.

Lastly, to assess whether taxonomic dominance (in terms of species richness at the family level) could be attributed to the evolutionary age of the taxon, we examined in the correlation between the age of family and the proportion of mOTUs. Family ages were obtained from TimeTree (http://timetree.org, beta version 5), with missing values obtained from major large-scale studies involving dating58,59,60,61,62. For the various statistical analyses, we excluded families with ≤10 specimens across all the samples and families that are present in one sample only.

Assessment of taxonomic neglect

To assess how much taxonomic attention has been given to the families dominating Malaise trap samples, the total number of species described for each of the top 20 families was obtained from CoL v22.3 (https://www.catalogueoflife.org/). For Bethylidae, CoL lacked information although the family was listed. We thus used values from a recent checklist instead63. For two families, we used the species richness in superfamilies (Noctuoidea and Platygastroidea). This was either because the family was not listed (Erebidae) or because of recent changes in family-level classification (Platygastridae, see above).

To next characterize the level of taxonomic neglect, we defined an NI as NmOTU/Nsp, where NmOTU is the number of mOTUs for a given family across the whole dataset and Nsp is the total number of species described as obtained from CoL. To evaluate potential drivers of taxonomic neglect, we next hypothesized that small-bodied or species-rich insect groups would be particularly prone to neglect, as being inconspicuous, poor in morphological characters and phylogenetically and/or taxonomically unwieldy (as due to their diversity alone). Large-bodied or species-poor families, we predicted, would be considered charismatic, accessible to morphological assessment and phylogenetically and/or taxonomically more clear cut. To test for such impacts, we modelled ln(NI) as a function of the mean body size of the insect family and the diversity of OTUs. To obtain diversity of OTUs, for each of the top 18 families and 2 superfamilies, species delimitation was conducted independently using objective clustering. The body range size limits for the calculation of mean-of-logs body-size was obtained from Rainford et al.64. For Crambidae, we used body-size range of Pyralidae, given that the study included Crambidae within Pyralidae. Similarly for Aleyrodidae, we used the body-size range for Aleyrodoidea. For Erebidae, the minimum and maximum forewing length was based on the combination of Lymantriidae and Arctiidae. For Platygastridae, it was based on combination of Platygastridae and Scelionidae. We modelled the relationship between neglect, species diversity and body size as ln(NI) ~ ln(NmOTU) × mean-of-logs body-size. Since we detected no interaction between ln(NmOTU) and mean-of-logs body-size (main dataset: coefficient ± SE: = −0.305 ± 0.227, t = −1.342, P = 0.198; expanded dataset: coefficient ± SE: = −0.141 ± 0.434, t = −0.324, P = 0.75), this term was removed from the final model (ln(NI) ~ ln(NmOTU) + mean-of-logs body-size).

We next examined the taxonomic attention given to the top 20 taxa over time. To this aim, we counted the number of species descriptions in Zoological Record. All data for the top 20 families were downloaded by search term ST = [Taxon name], as encompassing 181,985 studies. Studies in the past four decades (1980–2019) that describe species were identified by the sp nov epithet in the organism field. This approach identified 16,362 studies. The information on organism was then extracted to obtain the species and the family name, along with information on the year of publication and the authors involved. The data were then parsed to obtain the total number of species described in the study. For Erebidae, Crambidae, Aleyrodidae and Platygastridae, we assessed information at the level of the superfamily.

To test for a change in the relation between species diversity and neglect over time, we tested for an interaction between NI and Decade. To this aim, we compared two analysis of covariance models fitted to the univariate data: lm(log(Nsp10) ~ log(NI), data = UnivariateData), and lm(formula = log(Nsp10) ~ log(NI) × Decade, data = UnivariateData). Here, Nsp10 is the number of species described in a decade, with decades being 1980–1989, 1990–1999, 2000–2009 and 2010–2019. The fit of the two respective models was then compared by ANOVA (anova (model1,model2)).

Lastly, to further evaluate whether taxonomic work dedicated to the top 20 families has changed over time, we extracted information on the number of authors highly dedicated to a particular family, as scored from the number of authors exceeding a particular threshold (Sthreshold) of species descriptions. Given that some of these descriptions involved multiple authors, for each author i, the score was calculated as Si = \({\sum} {\frac{1}{{N_{{\mathrm{auth}}\_j}}}}\), where \(N_{{\mathrm{auth}}\_j}\) is the number of authors in the study j. For an article in which two authors described a species, each author would thus get an author score of 0.5 for this species. The number of highly dedicated authors was then scored as the number of authors with Si > Sthreshold per decade, where Sthreshold = 50.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.