Discrepancy of particle passage in 101 mask batches during the first year of the Covid-19 pandemic in Germany

During the first wave of Covid-19 infections in Germany in April 2020, clinics reported a shortage of filtering face masks with aerosol retention> 94% (FFP2 & 3, KN95, N95). Companies all over the world increased their production capacities, but quality control of once-certified materials and masks came up short. To help identify falsely labeled masks and ensure safe protection equipment, we tested 101 different batches of masks in 993 measurements with a self-made setup based on DIN standards. An aerosol generator provided a NaCl test aerosol which was applied to the mask. A laser aerosol spectrometer measured the aerosol concentration in a range from 90 to 500 nm to quantify the masks’ retention. Of 101 tested mask batches, only 31 batches kept what their label promised. Especially in the initial phase of the pandemic in Germany, we observed fluctuating mask qualities. Many batches show very high variability in aerosol retention. In addition, by measuring with a laser aerosol spectrometer, we were able to show that not all masks filter small and large particles equally well. In this study we demonstrate how important internal and independent quality controls are, especially in times of need and shortage of personal protection equipment.


Results and discussion
Setup and validation. The three international standards used for PPE certification require a flame photometer for particle passage determination. This device monitors the overall passage as total mass summed up across all particle sizes, which is considered sufficient as a lethal dose reference. In contrast, a laser spectrometer gives information on the number of particles passing the sample as a function of particle size. By mass-weighted averaging of passing particles, redundant information to a flame photometer can be acquired. The auxiliary individual particle size data gained via the laser spectrometer is displayed in the supplementary information attached to this manuscript. Additionally, information on the dependency of passage on particle size allows for further statistical analysis of mask performance. The laser spectrometer is used to extract the efficiency in particle removal for single particle size classes, as the international standards vary in the monitored particle sizes. However, the European Union decided to regard the respective international standards as equivalent to the FFP2 standard 49 . This is why all masks tested in the scope of this study have been subject to a unified testing procedure. As the use of a laser spectrometer is a deviation from the designated standards, in this study, we, first, use a flame photometer and a laser spectrometer in redundancy for method validation and, second, investigate the size-dependent protection performance of different mask batches.
To validate the standalone use of the laser spectrometer, a flame photometer was used in parallel to quantify the aerosol passage. The discrete measurements of the laser spectrometer were converted to an overall mass passage value to allow comparability and validation described in the "Methods" section. Figure 1 shows the overall passage of both detectors plotted against each other for validation. Ideally, both detectors measure the same passage and therefore all values would lie on the bisecting line.
The agreement between the two measurement methods is assessed with Lawrence Lin's concordance correlation coefficient ρ c 50,51 (see Eq. 4). The coefficient equals 1 if all points lie on the bisecting line and decreases for increasing deviations from this line. For the full data range (Fig. 1a) from 0 to 100% a ρ c of 0.9987 is obtained. In the clinically relevant range for protective face masks of up to 10% passage (Fig. 1b), a value of 0.95 is observed. This indicates a good agreement between the measurements by flame photometry and laser spectroscopy. Therefore, we consider both methods equivalent.
Filtration performance. During the first month that the pandemic hit Germany, the setup was established, and performance evaluation of different face masks was started to help communal organizations provide safe equipment to their staff. Therefore, our results indicate changes in the overall mask quality over the course of the pandemic. After the slackening of the pandemic situation, we used the acquired base of data for statistical evaluation to help evaluate mask protection performance and enhance product quality tests in the future. Figure 2  www.nature.com/scientificreports/ had production dates spanning the entire measurement period, the tested KN95 grade masks were produced only between February 2020 and June 2020, resulting from a rapidly increased availability during that period. The mask performance results are interpreted in close temporal relation with the course of the pandemic, increase in production capability, and mask availability. The strong decrease in mask quality starting February 2020 correlates with the significant increase in demand during the first Covid-19 wave and the following increase in production. Our results show that the increased production was accompanied by a decrease in mask quality. For KN95 masks, our results are limited to the production period from February 2020 to June 2020 and do not allow us to draw conclusions on quality beyond that period. The limited availability of FFP2 masks during that time led to an increase in the use of KN95 masks. KN95 masks were gradually superseded by FFP2 masks on the German market as soon as their availability stabilized.
Most tested FFP2 batches comply with the specification over the whole observation time. However, between February 2020 and June 2020, which coincides with the first wave of the pandemic in Germany, the required specification is not met by about 50% of the batches, with one batch being completely out of specification (> 70%). In contrast, most KN95 mask batches (92%) do not comply with the 6% passage specification. Hence, our results show a strong dependence of mask performance on the temporal market situation.
Irrespective of the mask production period, the mask performance was measured for each batch individually. The individual mask batch results of particle passage are displayed in Fig. 3. The passage of particles is displayed as violin plots, which are similar to box plots, except that their thickness at different passages reflects  www.nature.com/scientificreports/ the probability density of the data. In all violin plots, the passages are averaged over all particle size classes individually detected by the laser aerosol spectrometer, allowing for a conclusive evaluation of the spread between individual masks. This way, the data displayed here is consistent with the data assessment in standardized mask certification. The respective standardized maximal passage is indicated by a dashed line, while the axis is shrunk for passages above 10%. In Fig. 3a, passages of all investigated mask batches officially certified as FFP2 are plotted. The violin plots allow for an evaluation of the spread of passage between individual masks in the measurement. Strong deviations in qualities can be observed between the different mask batches. In general, three main groups of masks can be identified. First, masks that show a passage of particles well below the standardized limit of 6% (batches [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20]. This excellent product performance allows for statistical deviations while still meeting the standard for every single mask. Second, a group of masks that shows particle passage in the range of the standardized limit can be identified (batches [21][22][23][24][25][26][27][28][29][30][31][32][33]. Most products of this group are featured by high variability in passage between individual masks tested. This effect of high performance deviations between masks demonstrates a significant discontinuity in product quality among single masks. With less than 20 masks being tested in the standardized certification process, there is a high statistical risk of masks being certified that exhibit particle passages higher than the maximum allowed passage. Hence, the variability in product performance for some product types underlines the need for representative sample sizes to evaluate the product quality on a statistically reliable basis. Third, a group of masks can be identified that show particle passages way above the allowed limit of 6% (batches [34][35][36][37][38][39][40][41][42][43][44][45][46]. Figure 3b shows the violin plots of particle passage for the masks according to the KN95 standard. In contrast to the results shown in Fig. 3a, no masks with particle passages well below the standardized limit of 6% were found. In the group with the best particle retention performance, masks that show particle passage in the range of the standardized limit can be identified (batches [47][48][49][50][51][52][53][54][55][56][57], showing high deviations between individual masks. A second category of KN95 masks was found with particle passages in the order of 9% (batches 58-69). The other KN95 masks can be clustered in a third group that exhibits very high particle passages (batches 70-84). However, it must be noted that the KN95 products investigated in this study were all produced in the first period of the pandemic, in which also many products certified by other standards were not able to meet the standard's requirements.
The particle passages of N95 and FFP3 mask batches are displayed in Fig. 3c, d, respectively. For N95 products, most investigated batches show particle passages widely spread around the standardized limit (batches 86-90).
Only one batch of masks shows passages distinctly below the standardized limit of 6% (batch 85), and one batch stands out with a passage of more than 10% (batch 91). For FFP3 products, two of the tested batches exhibit very low particle passages (batches 92-93). Half of the as FFP3 certified mask batches show comparatively high spreads below or slightly over the standardized limit of 1% (batches 94-98). A third group of products, again, is featured by high particle passages of many times higher than the allowed maximum (batches 99-101).
The certification standards require the masks to pass a mass fraction criterion of the impinged aerosol, which is in accordance with the current understanding of toxicity referring to a certain dose 52 . The analysis of the total mass, in turn, reduces the influence of small aerosol particles on the measured performance result. Hence, masks only retaining large particles might still pass the criterion, while the passage of small particles might be well above the limit. This certification design is suitable for PPE applications in medical environments, where a maximum virus load must be prevented. In this context, the total amount of virus load scales with the aerosol particle volume. Hence, a mass-weighted protection performance evaluation is a good measure for effective human protection. In mask applications for environments containing high loads of fine particulates, however, particle passage sensitivity towards smaller particle sizes might cause severe differences in protection performance. In these cases, smaller particles tend to be of an elevated hazardous potential 53 . Hence, the retention of small particles in these use cases is of higher importance.
To investigate this crucial mask performance parameter, we used the particularity of the laser aerosol spectrometer of measuring particle passages as a function of the particles' diameters. For statistical evaluation of the size-dependent spread in retention, we calculated the standard deviation between all measured passages of different particle size classes. We determined the particle classes' passages by averaging the respective size class passages over all masks contained in one batch. In Fig. 4a, high median passages indicate low performance at most particle sizes. A small relative standard deviation of particle passage reflects a uniform particle retention behavior over particle sizes. In contrast, a high relative standard deviation of particle passage implies an increased spread in the passage for different particle sizes. In these cases, the measured particle size distributions indicate that the mask still retains large particles while smaller aerosols pass through.
Concerning particle passage deviation for different particle sizes, three categories can be identified. Most favorable are masks of category A, featured by a low median passage and low relative deviation of the particle passage across all investigated particle sizes. These masks uniformly retain particles and aerosols across the whole measured particle size range well below the standard's limit. Masks of category B show a low median passage but high deviations as a function of particle size. These products tend to retain larger particles while being more permeable to smaller ones. Masks of category C are least favorable as they show a high median particle passage. The low values for the relative deviation of passage show that these mask batches tend to be permeable across the whole span of particle sizes investigated.
In Fig. 4b, the particle median passage for all samples of one product batch and over all particle size classes is plotted over the highest particle passage of all particle size classes within one batch (max. particle passage). This depiction allows for a distinction between masks of three types; (A) masks that comply with the maximum passage for all particle size classes (median and maximum passage below the standardized overall maximum passage); (B) masks that show higher particle passages for some particle classes, but fulfill the required product quality of a maximum overall passage; and (C) masks that do not meet the standard's requirements.

Conclusion and outlook
Face masks are an effective tool to protect humans from aerosols that potentially contain viruses or other harmful substances. Depending on the desired application, different standards according to country of origin and product quality are defined. With the rise of the Sars-CoV-2 pandemic, the demand for face masks drastically increased in early 2020. Most severe was a scarcity in face masks of higher protection classes such as FFP2, KN95, or N95. The drastic increase in face mask demand led to a significant expansion of face mask production capacity. However, the standardized certification process does not require a renewed certification after such process capacity adaptions. As a result, incertitude evolved for end customers and raised the need for extensive studies on face mask protection behavior. In this study, we investigated 101 batches of face masks by various manufacturers. For protection performance evaluation, we used a setup complying with the defined standards for testing filtration performance during the face mask certification process. Our setup can be simplified by replacing cost-intensive equipment by low-cost alternatives and can thus be easily replicated with little financial resources. The aerosol generator could be replaced by ambient particles and the laser aerosol spectrometer could be exchanged for simple Rayleigh scattering detectors or a clean room particle counter.
Comparing the filtration performance of face masks on average over the production period, a tendency of lower protection performance can be identified for FFP2, KN95, N95, and FFP3 face masks produced in the early phase of the pandemic. However, all face masks investigated with a production date in the second half of 2020 or beginning of 2021 (FFP2, N95, FFP3) meet the standards according to their certification. Overall, tremendous differences in mask performances were observed in all protection classes. Only about a third of all mask batches investigated met the standard requirements, while another third showed particle passages slightly over the standardized maximum passage. In turn, this means that a third of all mask batches investigated in this study exhibited a particle passage significantly higher than the required standard, indicating a severe lack in protection efficiency.
Further, using a laser spectrometer in our setup allowed for an additional evaluation of the particle passage as a function of applied particle size. This analysis revealed additional differences in product quality among FFP2, KN95 and N95 mask batches. While some mask types retain all sizes of particles equally well, other mask types show a decreased particle retention for small particles. While this complies with face mask applications for medical environments, it might be undesirable for applications in environments with high loads of fine particulates.

Materials and methods
Experimental setup. In this work, an experimental setup based on DIN EN 149 and DIN EN 13274-7 is used to test all presented face masks. The setup consists of three main parts: an aerosol generator (A), a test chamber (B), and an aerosol analyzer (C), as shown in Fig. 5.
To test face masks concerning their aerosol passage, a defined aerosol is necessary to get comparable results consistently. A test following DIN EN 149 18 requires two different test aerosols, a sodium chloride aerosol and a paraffin oil aerosol. The sample needs to be exposed to the aerosols at an airflow rate of 95 L min −1 . To test a high amount of masks with a simple test and with the actual use of the mask known, in this work, only sodium chloride aerosols were used to determine the aerosol passage of face masks. The precise protocol for the aerosol passage test with sodium chloride and, thus, the requirements for the aerosol generator are stated in DIN EN 13274-7 54 . To obtain the required aerosol with a median in particle size distribution between 0.06 and 0.10 µm , an aerosol The generated aerosol is fed into a test chamber (see Fig. 5B). The test chamber needs to be closed against the environment and must provide an appropriate and sealed fixation for face masks. The airflow needs to be withdrawn from the test chamber behind the face mask, and the withdrawn air needs to be fed to suitable analyzers. In this work, a 'Smart Store 45L' (Orthex Group, Lohja (FI)) storage box is used as a test chamber. The storage box comes with a lid and is stable and sealed when a vacuum of up to 20 mbar under ambient pressure is applied. All tubing is connected to the test chamber with suitable IQS fittings. To perfuse the tested face mask with the required flow rate of 95 L min −1 a vacuum pump behind the sealed face mask withdraws the air from the test chamber. A 'WD 5 P' (Kärcher SE & Co. KG, Winnenden (DE)) is used in this work. The complete air needs to pass through the mask to leave the test chamber. The tests are conducted with a pressure difference between − 3 and − 7 mbar compared to ambient pressure. The pressure difference between the inside of the test chamber and ambient pressure is monitored by a differential pressure sensor (105). A second differential pressure sensor (104) measures the pressure drop of the face mask.
The test chamber needs to be divided into two parts by the face mask: the part in front of the mask, where the test aerosol is fed into, and a part behind the mask, where the air is withdrawn from the test chamber and fed to the aerosol analyzer to measure the aerosol concentration. To avoid leakage of particles, the mask needs to be tightly sealed. A mask mount was specifically designed to ensure tight sealing of the mask and for easy operation during sample change (see Fig. 5D).
The mask mount consists of a stable, leak-proof 3D-printed frame (Polyjet Objet Eden 260V 3D printer by Stratasys, Minneapolis, US) and a silicone inlay. The inlay was produced by using the frame as a mold. The face mask is clamped between frame and inlay for tight sealing during the measurement. The design of the mask holder ensures an identical filter area for all masks in one batch. To account for different face mask shapes, slight deviations (aspect ratio and circumference) from the original design were fabricated to ensure identical filtration area and tight sealing. Thus, the surface area of each mask during the particle passage test was proportional to its surface area. Every mask mount was once validated against a measurement of a circular, punched-out filter   www.nature.com/scientificreports/ sample in an o-ring sealed stainless steel filter holder. During a measurement, the pressure difference across the mask was measured. If the pressure difference was close to zero, the fit of the mask in the holder was checked and the measurement was repeated. We consider this measurement method as an improvement compared to the standard procedure of the German regulatory authority, as a fixed, circular surface area for the test of the filtration performance of each mask, independent of the surface area of the mask is used.
To analyze the aerosol concentration, DIN EN 13274-7 states an aerosol flame photometer with sample ports in front of the face mask and behind the face mask. In this work, a 'FP 8400' (Krüss Optronic GmbH, Hamburg (DE)) is used (see Fig. 5C). The flame photometer measures the concentration of sodium chloride in a gas stream by combustion in a flame. The flame is created by the combustion of propane with filtered and compressed air. Photo diodes detect the color change of the flame due to the combustion of sodium. The aerosol-containing sample flow is withdrawn from the tubing behind the test chamber by a membrane pump (KNF Neuberger GmbH, Freiburg (DE)), as shown in the right part of Fig. 5, and fed to the flame photometer. Furthermore, a laser aerosol spectrometer (LAS) '3340 A' (TSI Inc., Shoreview (US)) is used for aerosol concentration measurements (see Fig. 5C). The LAS counts particles flowing through a measurement volume by measuring light scattering induced by particles crossing a laser beam. Thus, the LAS measures the number of particles and the related particle size distribution in the sample flow. The used LAS measures 99 particle size classes of 4.5% class width between 90 nm and 7.5 µm . The LAS was connected to the outlet tubing of the test chamber, next to the sample port for the flame photometer. The LAS is equipped with a sample pump. Thus, no additional pump is needed.
Experimental method. In the experimental setup, two measurements are necessary to evaluate the filtration performance of a face mask. The first measurement is a reference measurement, obtaining information on the aerosol in the test chamber. Therefore the mask mount is placed in the test chamber without a face mask installed. This measurement is also used to validate the generated aerosol and particle concentration in the test chamber with DIN EN 149. During the measurement, the test chamber is closed. The aerosol concentration and particle counts over one minute are measured for five consecutive minutes using the flame photometer and LAS. The first two minutes of each measurement are excluded from the evaluation of the mask's filtration performance. As flow rates are slow compared to the length of tubing, this time is needed to reach equilibrium in the whole system.
After the reference measurement, the experiment with a mounted mask is performed. The sample holder is placed in the test chamber after checking for tight sealing of the mask. The filtration performance of the face mask is then calculated by using the collected data of minutes three to five. For evaluation of the filtration performance, the test measurement is compared to the reference measurement. When using the LAS, a correction term is used to calculate mass concentrations of the aerosol. This calculation procedure is described in the next section.
Data evaluation. The detected mass of particles m i per particle class i is calculated from the LAS data by averaging the particle count N i per size over three minutes multiplied by the individual particle mass normalized by the mean sample volume flow V sample : Here, the particle mass is calculated as the particle's minimum volume with diameter d min . The density ρ cancels out when calculating the particle passage per size: The variables m filter,i and m reference,i are the particle mass per size with and without an installed filter, respectively. For the comparison to the results from the flame photometer, the mean particle passe is calculated as: The concordance correlation coefficient 51 between two measurement methods x and y is calculated as where s x and s y are the variances of the data set x and y and s xy is the covariance. x and ȳ are the arithmetic means.
Statistical evaluation was done in Python using pandas 55,56 and seaborn 57 , visualizing the results in Matplotlib 58 . Violin plots were created with the standard seaborn implementation. The white dot in the center of each plot is the median of the measurement series. Surrounding the white dot is a small boxplot chart with a black rectangle showing the end of the upper and lower quartiles. The adjacent black line extends from the minimum to the maximum excluding any outliers. Around the boxplot diagram the blue the kernel density plot is shown, which represents the distribution of the measurements. The wider the kernel density plot, the more measurements are in this region, with each data point equally weighted. The number of measurements per batch is shown on the X-axis in parentheses.