Introduction

Next Generation Sequencing (NGS) technologies have dramatically changed the field of genomics and are routinely applied to a variety of functional genomics investigations including, but not restricted to, whole genome sequencing, global identification of genomic rearrangements, epigenetic modifications, single nucleotide polymorphism (SNP) discovery, transcriptome profiling and metagenomics. In recent years, using these technologies thousands of genomes assembled from short DNA sequence readings of humans, plants, animals and microbes have been collected and explored, enabling scientists to develop a deeper understanding and gaining new insights into the molecular mechanisms related to different diseases, including many types of cancer, allergies, or other disorders (Wiese et al., 2018). Furthermore, the genomics has profoundly influenced the pharmaceutical industry and reshaped the processes allowing to discover, investigate, and develop new drugs. No less important is the research carried out using these technologies in the environmental field for industrial and biotechnological purposes (Tiwari et al., 2018). Indeed, NGS is a complex process that, on the one hand, requires the preparation of sequencing libraries that respond to the specific standard requirements of the platforms, and on the other hand generates unprecedented volumes of data to be analyzed (Stephens et al., 2015). Moreover, an NGS analysis usually involves collaboration between several departments, laboratories and data analysis groups, characterized by different scientific backgrounds and, above all, applying different experimental approaches. With the growing need of managing information, it has become challenging to keep track of data, processes and outcomes of research over long periods of time and across the collaborating units. Nowadays Big Data generation and its management create extraordinary challenges for storage, transfer, analysis, interpretation and last but not least security of information. Regarding this last aspect, the scientific community has recently expressed the necessity to manage NGS data according to the principles of Findability, Accessibility, Interoperability, and Re-usability of digital assets (FAIR data) (Wilkinson et al., 2016; Corpas et al., 2018). The expectation is to produce digital resources with more rigorous management and stewardship that can be used by the entire scientific community. Good data management is not a goal in itself, but rather it is the key conduit leading to discovery and innovation, through data integration and reuse by the scientific community after the publication process. Good data management, ensuring reliability and usability, needs a holistic approach tracking the process of data and metadata generation and all the different organizational aspect that, on the one hand may affect it, and on the other hand can keep it under control. This can be achieved by means of a management system focused on the quality of the results.

Furthermore, in recent years, in the context of the scientific research, we are witnessing a new phenomenon, defined as “reproducibility crisis” by Baker (Baker, 2016) and Dirnagl et al. (Dirnagl et al., 2018) characterized by the reduction of the reliability, reproducibility, traceability and predictability of research results. These problems can not only compromise the robustness and rigor of research (Dirnagl et al., 2018) but have also a significant impact from an economic point of view, reducing the profitability of research funds (Lanati, 2018). The standardization and simplification of experimental workflows, such as those applied for “omics” applications, is becoming a need both for academic and private research laboratories. As described by Endrullat et al. (2016), standards act as basic guidelines to ensure comparability and exchange of experimental data conducive to the acceleration of the innovation process, aiding improvement of transferability, transparency and reproducibility of results. Furthermore, the advantages deriving from the standardization of processes could reduce costs and increase services (Endrullat et al., 2016; Cargill, 2011). A Quality Management System (QMS) can support the correct management of the NGS research environment, providing directions for data and operations management. A suitable quality system ensures safety, reliability and reproducibility of the non-clinical tests on chemicals intended for use on humans, animals and the environment. A QMS supports the generation of high quality scientific data and associated services, it is also helpful in improving the economic and social impacts of research. Quality research management reinforces scientific communities and improves the attractiveness and effectiveness of the service. In a QMS activities are properly planned and documented, operations are regulated by means of standard operating procedures, and the correct behavior, compliant with internal and external standards, is guaranteed by regular inspections. A QMS can help in giving proper attention to sensitive data and in correctly managing them, setting internal standard, organizing rules and forms and maintaining due control. Good Research Practices, as a quality management standard dedicated to the research environment, are at present not yet organized in an international reference text and consist of different prescriptive documents that are drafted and/or personalized by each research institution interested in aspects of quality management. Researchers can only refer to the WHO Handbook of Quality in Biomedical Research (WHO, 2006) as a guiding text to comply with generic quality principles. However, several references for designing a QMS can be found among international standards suitable for the management of a research laboratory: ISO 9001:2015, the most general quality management standard; ISO 17025, derived from the ISO9001 and dedicated to test and calibration laboratories; and the Good Laboratory Practice, mandatory international reference for development and testing of drugs and other substances intended for human and animal use (Lanati, 2018).

In this paper, we describe the choice, the design and the application of a QMS, called ReOmicS (Research Environment management system for Omics Sciences), at the Molecular Biodiversity Laboratory (MoBiLab), a NGS research infrastructure located in Bari (Italy) at the Institute of Biomembranes, Bioenergetics and Molecular Biotechnologies of CNR (CNR-IBIOM). MoBiLab is a research environment, fully equipped with operative platforms based on the most innovative NGS technologies and powerful resources for data storage and computational analysis, whose mission is to contribute to innovation with original studies. Moreover, CNR-IBIOM is involved in the construction of the national nodes of ELIXIR and LifeWatch Research Infrastructures (included in the ESFRI Roadmap), as well as in a substantial empowering of its infrastructural components for omics data production and analysis consistently scaling up the available instruments and facilities. The described experience could represent a scalable model to be applied to MoBiLab and to other research laboratories in order to ensure the highest levels of reliability, reproducibility and traceability of the results, a process that is also expected to foster their potential exploitation.

Material and methods

Considering the managerial aspects of the study, methods described in this section are tools used in quality and organizational management, occasionally modified to be adopted for the specific use of a research laboratory, such as MoBiLab.

Decision grid

The decision grid (or matrix) is a tool that supports a decision among many options. Once the aim of the decision is clearly defined, the criteria used to characterize each solution must be identified. Each criterion is given a weight (1 = lowest to 5 = highest) based on its importance in the final decision. The selected options among which the choice has to be made is then assessed with respect to their suitability to each criterion (1 = lowest to 5 = highest). The sum of the weighted assessments gives the final score for each proposal. Referring to Table 1, the head of the table defines the aim of the decision. The options are listed in the columns. Criteria are listed in the rows and weighted in importance in column W. Each option is given a specific assessment (column A) with respect to the relative criterion, and an overall score, which is the product of the importance of the criterion and the given assessment. The final score for the proposal is given by the sum of all scores for each criterion.

Table 1 Results of the decision grid
Full size table

SWOT analysis

The SWOT analysis represents, within a rationale, the influence exerted by some key factors on a goal in order to identify actions that reinforce the positive factors and counteract the influence of negative factors. The key factors in the analysis, whose initials give the name to the technique, are:

  • Strength: a resource that can be used to best achieve the goal;

  • Weakness: an obstacle to achieving the goal;

  • Opportunity: a favorable situation in the external context that favors the achievement of the objective;

  • Threat: an external, unfavorable situation in the external context that potentially hinders the achievement of the objective.

The analysis combines internal factors (strengths and weaknesses) and external factors (threats and opportunities), as well as positive aspects (strengths, opportunities) and negative aspects (weaknesses, threats). In this way SWOT Analysis allows defining strategies aimed at capitalizing on strengths, eliminating weaknesses, exploiting opportunities and mitigating threats.

Risk assessment

A risk assessment was performed on the main analytical process, according to the requirements of an ISO High Level Structure (HLS) and the ISO 9001:2015 standard. For each experimental step (Fig. 1, first column), some pitfalls were identified (second column); each pitfall was assessed with respect to its Severity S (from 1 = low to 3 = high) i.e., how serious would be the consequence of an error on the final result, and Probability P (from 1 = low to 3 = high), i.e., how frequently a specific mistake has recently occurred (column 3 and 4, respectively). The Risk R in column 5 was then calculated as S × P for each pitfall identified. Risk values range from 9 (greatest) to 1 (lowest). Operations with risk R greater or equal to 4 are judged worthy of specific interventions to prevent errors, as recorded in column 6 “Solution”. Interventions are prioritized according to the level of risk. Referring to the legend of Fig. 1, colors indicate the need for improvement actions: red for urgent, orange for medium and yellow for minor need, while green indicates no need for action.

Fig. 1
figure 1

Risk assessment of the primary process at MoBiLab before the introduction of the ReOmicS. Pitfalls are placed in descending order according to R. Colors indicate the need for improvement actions: red for urgent, orange for medium and yellow for slight need, while green indicates no need for action

Full size image

SIPOC

The SIPOC diagram, first outlined by Juran (Defeo and Juran, 2010) calling it TRIPOL, was then employed in the Six Sigma approach for analyzing a process. It is named SIPOC from the acronym of Supplier, Input, Process, Output, and Customer: the key elements of a process. A flowchart of the process is usually inserted in the third column “process” and for each step: input supplier, input needed by the operation, output of the operation and recipient of the output (customer) are listed.

The SIPOC-like flowchart is structured on the following categories:

  1. 1.

    source: the origin of the input

  2. 2.

    input: raw data, metadata, materials or samples needed by the study activities

  3. 3.

    process: steps of analysis and controls, logically linked

  4. 4.

    supervisor/person in charge: the supervisor and/or the person in charge to carry out each task of the previous column

  5. 5.

    output: the result/product of each task

  6. 6.

    procedure: the SOP describing the specific task

Metrics

Due to the limited availability of data for the period preceding the introduction of the QMS, the metrics system has been necessarily simplified to two indicator of efficiency and three indicators of effectiveness:

Fig. 2
figure 2

Efficiency indicator: analysis throughput. The values of four parameters (total number of processed samples, the average number of samples, the number of sequencing runs and the sequencing platform output), related both to all the MoBiLab applications (a) and only to Metagenomics (b), referring to the 3-year periods (2013–15 and 2016–2018), are shown

Full size image

  • efficiency:

    • outcome of the updated risk assessment, compared with the initial one performed in designing the QMS. With respect to the initial Risk Assessment, three new columns have been added: “New-Probability” with the updated probability of the pitfall(s) considered, “New risk Assessment” with the updated value for the parameter R = SxP, and “Audit” recording reasons and considerations regarding the improvement. A paired one-sided Wilcoxon test was performed, to verify whether the risk estimation prior to the QMS adoption was significantly higher than the next.

    • evaluation of the analysis throughput in terms of the total sequencing run number, the total number of processed samples, the average number of samples per sequencing run and the run output (Gb). For each parameter, the values related to the two three-year periods were collected. The analyses were carried out on both the data derived from all the MoBiLab applications (Genomics, Transcriptomics and Metagenomics) (Fig. 2a) and those produced only from Metagenomics (Fig. 2b).

  • effectiveness:

    • number of publications and the related impact factor (base: 3-year period): we considered the peer-reviewed publications of three researchers 100% involved in MoBiLab research projects (Source: JCR-ISI Web of Knowledge; https://login.webofknowledge.com). In case of co-authorship the journal was counted once, and the number of publications within the period has been calculated considering the average number of the papers per year. The scientific areas of the journals are Biochemistry, Genetics and Molecular Biology Medicine Agricultural and Biological Sciences, Immunology and Microbiology, Multidisciplinary Environmental Science, Neuroscience, Computer science, Mathematics.

    • scientific attractiveness, i.e., number of active external collaborations in MoBiLab publications (base: 3-year period) (Source: PubMed-National Library of Medicine; https://www.nlm.nih.gov/bsd/pubmed.html): author’s affiliation to the papers published in the first 3-year period 2013–15 (before the introduction of the QMS) are compared with author’s affiliation referring to the 3-year period 2016–2018, after the progressive introduction of the QMS.

    • satisfaction survey: two separate surveys were prepared using the online tool SurveyMonkey (https://it.surveymonkey.com). The first (B9M5SNL), dedicated to all customers/collaborators about perceived quality, was sent by mail to 54 MoBiLab collaborators. The second (BDDF65F) was sent, in addition to the first, only to those customers/collaborators (25/54) who worked with MoBiLab in both three-year periods before and after the introduction of the QMS. A two week deadline was given. The analysis results were provided by the tool and further analyzed and elaborated by the team (Supplementary Material (SM) 1 and 2).

The indicators of efficiency measure the ability of the MoBiLab to increase productivity and reduce costs, while the effectiveness indicators show the quality and importance of the analysis of results.

Results and discussion

Choosing the QMS standard

To choose the best reference standard for the characteristics of MoBiLab, we compared three international standards: ISO 9001:2015, ISO 17025, and GLPs by means of a decision grid. The criteria for this choice were identified as:

  • compatible with regulation environment

  • suitable for customer’s requirements

  • oriented to R&D

  • focused on analytical process

  • low management costs

  • easy to fit in Laboratory activities

  • not linked to external third parties

  • suitable for expansion

The results are illustrated in Table 1. Evaluating the criteria for choice, we considered that the “customers” of the MoBiLab research services are laboratories already working under the principles of GLPs and they could benefit from a rigorous and standardized work environment for the production of their data, as well as from a common management language and references. All criteria are listed in the first column of Table 1. The GLP obtained the best assessment weighted on the importance of each criterion, mainly for their suitability for customer’s requirements, the lower cost, the independence from third party evaluations and the opportunity for development.

GLPs are mandatory in OECD countries for preclinical tests, but should be also considered as a reference for laboratory management systems, that can be referred to as “GLP-like” quality systems, although outside GLPs main scope. As textually described by Kauffmann et al. (2017) the application of GLPs principles to “omics” studies based on NGS, in a regulatory context, would serve the following goals (i) to promote the consistent quality and validity of data used for determining the safety of chemical products—a primary objective of the GLP principles (OECD, 1998); (ii) to promote transparent process descriptions and thus support the traceability of study results; and (iii) to facilitate the exchange of information and enhance the regulatory impact of ‘omics data, when successfully used for hazard and risk assessment purposes. The use of GLP standard system helps the management of the human genomic data sharing respecting the privacy of the data, the reuse of the data under the current legislation and any further integration.

Design of ReOmicS

A director, three researchers, a technician and a quality consultant have been the working group that has met fortnightly via videoconference. The team assessed the choice of the GLPs as main reference for ReOmicS by means of a SWOT analysis, whose results are illustrated in Fig. 3. Strengths and opportunities validated the choice of following GLPs as main reference against minor weaknesses and threats. In any case, as illustrated in what follows, some actions have been planned and taken in order to tackle some weaknesses and threats in a near future.

Fig. 3
figure 3

SWOT analysis diagram. Strength, weaknesses, opportunities and threats have been evaluated for the choice of Good Laboratories Practices (GLPs) as QMS reference

Full size image

The team then started the design of ReOmicS from the risk analysis of the main NGS process, based on a standard Risk Assessment. With this analysis, the team aimed at identifying the laboratory’s weaknesses in order to address them with specific interventions when developing the QMS.

Errors and problems experienced during the last 3 years were collected and attributed to the relevant process steps. Each problem was then assessed with respect to severity S and probability P. The risk R = S × P associated with each pitfall determined the need and the priority of a specific operating procedure. Figure 1 summarizes these results.

Major problems to be addressed when defining Standard Operating Procedures (SOPs) were identified in the fields of communication with the customer; traceability of samples; warehousing, archive, metadata and database management; planning; organizational structure; and controls and checks. No needs for specific instructions were envisaged for problems with R less than 4. Major needs and related priority were taken into account in the drafting phase of the SOPs.

As a first step, in order to define the internal context, the MoBiLab organizational structure was designed according to the GLP requirement taking into account the dimension and the constraints of the research institution: the major roles, such as Director, Study Director, Laboratory Manager, Principal Investigator, Archivist et al, were identified and assigned to laboratory staff (Fig. 4). Once defined the roles and responsibilities, the team analyzed the laboratory internal processes, identifying primary and support processes (see Table 2). The team outlined the main (primary) process by means of a Supplier-Input-Process-Output-Customer (SIPOC)-like flowchart, which includes the person in charge of the activity and the related documented information (SOPs, and records) (see Fig. 5).

Fig. 4
figure 4

Organizational Chart. Under the supervision of the MoBiLab director, resources are divided into two main groups: (i) managerial and technical support and (ii) research and experimentation

Full size image
Table 2 List of the primary, support processes and SOPs of ReOmicS
Full size table
Fig. 5
figure 5

SIPOC-like flowchart of the Study management. Columns collect inputs and related providers, main flow-chart with tasks, person in charge/supervisor for every task, output and prescriptive document (SOP) for each task. The flowchart is divided into two main sections: Execution/experimentation and Analysis/results

Full size image

This SIPOC-like chart acted as the backbone of the SOP of the management of the Study: moreover, most steps of the study process, together with the results of the risk assessment, led to the identification of technical/scientific procedures, these were then accompanied by attachments describing technical details as required. After having identified the SOPs required for the operational processes, researchers were provided with a template and with the instructions to draft them. In parallel, management procedures were defined by the whole team, drafted, and supported by flowcharts and other quality tools whenever needed (e.g., SIPOC). SOPs were ranked by priority, driven by the risk assessment results; few SOPs required by GLPs were not considered since were not needed for our specific research activities (e.g., management of test systems). The management support processes were described by SOPs and were, as far as possible, compliant with GLP requirements.

Following the results of the SWOT analysis (where this aspect was judged as a weakness) and in order to ensure the highest level of reliability, reproducibility and traceability of the results, the team also planned to develop and optimize a LIMS (Laboratory Information Management System) platform for managing all the laboratory activities through a suite of integrated modules, in collaboration with an Italian ICT company. The platform will be structured starting from the SIPOC-like flowchart for the management of the study and the modules will be developed and customized in agreement with the SOPs.

The structure of the SOP list (Table 2) is directly related to the allocation of responsibilities in the laboratory and conforms to the organization of the management of the studies (Fig. 5): the primary and the scientific SOPs are the responsibility of the researchers and technicians operating in the MoBiLab under the supervision of a Principal Investigator, nominated by the director of the study. At the same time, management and technical support SOPs govern staff indirectly involved in the project, caring for an environment suitable for the studies. The split into two different areas, research and support, is clearly represented in the organization chart (Fig. 4).

Indeed, the MoBiLab belongs to a public research institution whose mission is to achieve scientific outputs in national and international funded project. Staff organization is related to the skills required and tasks assigned in the study program. For this reason, in the primary SOP the study corresponds to the project and the director of the study refers to the scientist responsible of the project. Only in a few cases the MoBiLab is working as a service provider, producing genomic and data analysis directly commissioned by external customers. For all these reasons, the scientific SOPs can evolve by integrating new requirements highlighted by customers or scientific partners.

Following all these considerations, the ReOmicS was structured as illustrated in Table 3. Each SOP is structured according to a general template with the following sections:

  • -scope

  • -purpose

  • -definitions, terms and acronyms

  • -references

  • -activities and responsibilities

  • -materials and equipment

  • -procedure

  • -forms

  • -safety rules

  • -history of revisions (change register)

Table 3 ReOmicS quality management system
Full size table

Results of the application of ReOmicS

The system has been progressively introduced, starting in 2016 from the scientific SOPs. At present the application of ReOmicS is underway with few limitations, due to the fact that processes and their management are challenging to pursue without IT support. This is also the reason that a LIMS is being developed. For example, audits have been conducted in limited form, mainly focusing on the suitability and effectiveness of the quality management system. Despite these limitations, the MoBiLab, in the 3-years following the introduction of common rules and references, is experiencing a concrete improvement, as testified by the positive trend of the five metrics chosen: risk assessment, analysis throughput, number/quality of publications, external collaborations and satisfaction survey.

The first metric chosen for the assessment of efficiency is the comparison between the outcomes of the risk evaluation before and after the introduction of ReOmicS (Fig. 6). The results show a statistically significant improvement (p = 0.0005) with respect to the initial assessment for the application of technical SOPs, while that of management SOPs is still limited. A better compliance to the QMS is expected in the future, mainly in the areas of study-planning, assignment of tasks and storage control.

Fig. 6
figure 6

Efficiency indicator. Efficiency is evaluated by means of the update of the risk assessment performed in the initial phase of the project, before the application of ReOmicS

Full size image

A second metric has been chosen to evaluate the efficiency, i. e. the overall performance of MoBiLab, in terms of total number of processed samples, number of samples/sequencing runs, number of sequencing runs and sequencing platform output (GigaBase). The data shown in Fig. 2 compare the 3-year periods, 2013–2015 and 2016–2018. Considering all the MoBiLab applications (Genomic, Transcriptomics and Metagenomics), the analyzed parameters, except the number of sequencing run, show an improvement after the introduction of ReOmicS (Fig. 2a). At the same time, it is important to underline that, despite the reduction in the number of sequencing runs due to a forced six months interruption of MoBiLab activities for logistical issues, the total number of samples processed and the average number of samples increased together with the platform output. In an NGS analysis, we can speculate that maintaining a high throughput whilst at the same time increasing the number of samples, represents an important laboratory challenge. These results can be ascribed to operator’s competences and training, but also to the improvement of the management of the process and the control of the analysis provided by the QMS. The efficiency of the laboratory was therefore assessed by taking into consideration the amount of data produced, referring mainly to Metagenomics analysis, the most requested application at MoBiLab (approximately 52% of the total amount of analysis performed) (Fig. 2a). Overall, a positive increase was shown (Fig. 2b) by all the parameters during the 3-year period 2016–2018.

To assess the influence of the QMS on the effectiveness of the MoBiLaB, three metrics have been chosen. The first one is the number of publications and the related Impact Factor (IF). In Table 4, data referring to the first 3-year period 2013–15 (before the introduction of ReOmicS) are compared with data from the 3-year period after the progressive introduction of the QMS, 2016–2018. Table 4 also shows the average IF values obtained for each period. The lowest and the highest values of journal IF were excluded from the analysis. The QMS improved also downstream processes as demonstrated by the increased number of papers published in peer review journals. The number of published papers has grew from 13 to 23 in the last 3 years. Indeed, the number of published papers doubled even if the IF increase is not significant (data not shown). The second metric chosen for the assessment of effectiveness of ReOmicS, i.e., number of active external collaborations in MoBiLab publications (based on a 3-year period), shows the attitude of the laboratory to be a national and world leading scientific NGS laboratory and to be an enabling facility in the support of science. Table 4 shows the average number of external authors in the two 3-year periods. The total number of authors for each paper did not significantly change in the two periods considered, nor did the number of authors with an Italian affiliation. On the other hand, the number of general affiliations increased by a third and the number of international collaborations almost tripled.

Table 4 Effectiveness indicators
Full size table

The third effectiveness metric measures the satisfaction of customers and collaborators who had the opportunity to take advantage of the analysis service of MoBiLab, by means of two separate surveys: the first one dedicated to all customers/collaborators about perceived quality and the second to customers/collaborators who worked with MoBiLab in both three-year periods before and after the introduction of the QMS. The first survey was sent to 54 collaborators and 25 answers were collected. The second survey was sent to 25 collaborators, obtaining 12 answers. Of these last 12, 6 were discarded for inconsistency in answers to single questions, for this reason only a qualitative evaluation can be made. Results of both surveys are illustrated in Fig. 7 and show a good level of satisfaction from customers and collaborators, together with a demonstrable improvement of perceived quality after the introduction of ReOmicS.

Fig. 7
figure 7

Customer satisfaction. A qualitative measure of the customer satisfaction was evaluated analyzing the results obtained from the two surveys: the first one dedicated to all customers/collaborators about perceived quality (a) and the second to customers/collaborators who worked with MoBiLab in both three-year periods before and after the introduction of the QMS (b)

Full size image

As a final consideration on metrics, it was difficult to gather complete and detailed data regarding the projects developed in the years 2013–2015 for comparison with those pertaining to the years 2016–2018 because, before the ReOmicS introduction, a lot of the information was scattered among different research environments within and outside the MoBiLab. Since this data unavailability was judged unacceptable in maintaining due control on the work of the laboratory and on the improvement process, a more complete set of metrics has been studied which will be integrated in the planned LIMS (Table 5).

Table 5 Future process and product indicators
Full size table

Main deviations from GLPs

Not all the requirements of GLP can be accomplished in the development of ‘omics’ studies, as clearly shown by Kauffmann et al. (2017). The limitations involve technical aspects, but in our case have had an impact also on the organizational requirements.

The first requirement that cannot be met is the management of test systems, because in the NGS procedures they are not used. External databases are used as reference and these are validated by the well-known mechanism of peer review. This is in partial disagreement with the GLP direction about data management and validation, but is common practice in genomics.

As far as data storage is concerned, MoBiLab depends on the servers made available by the INFN. The commercial agreement with INFN is stipulated by the IBIOM Institute: so far MoBiLab is not in the position to insert specific GLP requirements. With the planned development of the LIMS, new conditions and agreements more suitable for GLP compliance about data management will be implemented.

As an example of GLP procedure requirements that needed a new definition, the compliance statement required by the GLP is intended not towards the GLP, but to ReOmicS QMS itself.

As an example of organizational GLP requirements that could not be met, the dimension of the research unit and specifically of the laboratory is an issue when trying to identify an independent quality assurance structure. However, within the laboratory, a person has been appointed for the quality assurance tasks described in the dedicated SOP with support from an external quality consultant for methodological matters or concerns. Furthermore, the title of Principal Investigator (PI), which in the GLP is an individual who, for a multi-site study, acts on behalf of the Study Director, is known here as “research project manager”, as in the most common meaning for research laboratories working on funded projects.

Moreover, the role of archivist has not yet been allocated, since the planned introduction of a LIMS will ease the task of archiving and will allow a clearer allocation of responsibilities.

Conclusions and future perspectives

We can demonstrate that the application of a QMS, giving precise references for research management, introducing controls—thus increasing both result reliability and reducing opportunities for error—and promoting the efficiency in planning, conducting, analyzing and reporting on the processes, represents a valid tool for overcoming these difficulties and, at the same time, an opportunity for significant improvement for a research laboratory.

In the experience illustrated, GLPs—among different quality management standards—was judged most suitable for the purpose of the MoBiLab QMS, when implemented in the aspects appropriate to the characteristics of the laboratory. The development of the QMS was performed making use of quality techniques and methods to ensure a lean and rigorous process.

Among the positive outcomes that can be ascribed to the adoption of the QMS, we can stress an evident increase in the efficiency of the laboratory, evaluated by the decrease of the risks and of the errors occurring during analysis workflow. Moreover, staff was more motivated thanks to a better organization of the team and an acknowledgment of their competences. The effectiveness was also improved, increasing the number of collaborators, the customer confidence and the availability of a databases organized for future investigations.

Indeed, the experience of QMS in MoBiLab demonstrates that the performances of the analysis and number of the primary “products” of academic research—publications—increases after QMS introduction, together with the appeal of the laboratory as witnessed by more active international collaborations. Number of publications in high-impact factor journals, number of citations, and number of opportunities of excellent scientific collaborations indicate how the laboratory aims to be a national and world scientific leading infrastructure and an enabling facility supporting science.

Experimental and service data (throughput analysis) prove that the laboratory, thanks to the ReOmicS introduction, is increasing its ability to provide high quality scientific data and associated services.

The results of the surveys give evidence of a positive user satisfaction regarding support and collaboration, not only in reference to the facility, but also to the staff employed in the MoBiLab. Good opinions were also expressed comparing MoBiLab with other similar laboratories. We have planned to run such surveys periodically with different audiences to assess satisfaction, achievements, collaborations and expectations.

As future perspectives, scientific, economic and technological impact could be assessed reporting, for instance, the training of skilled researchers, the development of new methodologies and software for NGS, the growth of network and social interactions or the creation of a new firm (e.g., a spin-off).

As proposed by the OECD (2019) for the assessment of Research Infrastructures, the metrics chosen to evaluate efficiency and effectiveness of MoBiLab before and after the introduction of the QMS can be presented as possible indicators to demonstrate the impact of an NGS research and analysis laboratory.

The quantifiable impacts captured through quantitative metrics (number of publications, citations…), as well as the non-quantifiable metrics obtained by dedicated surveys, are also helpful when approaching an economy and policy impact analysis of the genomic and bioinformatic research.

Although the ReOmicS QMS did not incorporate all requirements of the GLPs, and its application is still to be completed with respect to a small number of controls and the complete traceability of results, the positive outcomes already obtained are also due to the gradual increase in confidence by laboratory staff with the quality approach and to the early adoption of the standard protocols described by the technical SOPs.

This experience and the results obtained prove that a NGS laboratory, and therefore any other research laboratory, can benefit from the introduction of a quality framework, if properly translated from the generic standards and adapted to the specific requirements of a research environment. The metrics and the indicators showed in this study will be followed up in the implementation of the MoBiLab thanks to the grant received by the Italian Research Minister to empower facilities and equipment of this laboratory.

The standards of the ISO 900 family are often declined in sector-specific versions (e.g., the already mentioned ISO 17025 for testing and calibration laboratories, ISO/TS 16949 for automotive, ISO 13485 for medical devices), adding requirements unique to a particular application. The ReOmicS has this precise purpose: to detail the requirements for the management and control of a generic research laboratory. To this end, as a first choice, we took as a reference the GLPs, whose adequacy for a research laboratory in ‘omics—despite some specific exclusions—is attested by Kauffman et al. (2017). The directions of the GLP are about management and control of the study and experimentations (the primary process), and of the tools and materials used, as well as the rules for reporting and archiving data, samples and documents (which are only a few of the support processes envisaged by the ISO9001 standard). In a general sense, these aspects are common to all research laboratories, regardless of the field of application. Starting from the structure of a GLP-like QMS, the ReOmicS can be completed with the specific requirements of ISO 9001 on the parts not governed by the GLP, such as context and risk assessment, supplier management and improvement process. In this light, we strongly believe that the ReOmicS can be taken as a reference for any type of research laboratory.

Indeed, the ReOmicS GLP-like quality system is expected to evolve into a complete quality system according to ISO9001 to achieve the specific certification. For this reason, a risk assessment—both strategic and operational has already been performed, and several SOPs have already been arranged to comply with main requirements of ISO 9001.

In the future, quality management would be streamlined by the introduction of the IT tool (LIMS). This development is expected to foster the potential exploitation of the NGS activities of the MoBiLab. The evolution of the ReOmicS is then expected to follow the Lean Production (a.k.a. Toyota Production System), which is a wide-ranging methodology developed in manufacturing to reduce waste and improve product quality (Womack et al., 1991) and recently used also in the research environment (Barnhart, 2013). The Lean approach will be strictly connected with the LIMS system, leveraging its features to ensure the best control. The GLP-like system and the Lean approach will allow the MoBiLab to improve its efficiency, limiting wastage of time and materials, and reducing opportunities for error, at the same time enhancing the effectiveness traceability and reproducibility of results.