Main

Artificial intelligence (AI) methods have the potential to revolutionize the domain of medicine, as witnessed, for example, in medical imaging, where the application of computer vision techniques, traditional machine learning1,2 and—more recently—deep neural networks have achieved remarkable successes. This progress can be ascribed to the release of large, curated corpora of images (ImageNet3 perhaps being the best known), giving rise to performant pre-trained algorithms that facilitate transfer learning and led to increasing publications both in oncology—with applications in tumour detection4,5, genomic characterization6,7, tumour subtyping8,9, grading prediction10, outcome risk assessment11 or risk of relapse quantification12—and non-oncologic applications, such as chest X-ray analysis13 and retinal fundus imaging14.

To allow medical imaging AI applications to offer clinical decision support suitable for precision medicine implementations, even larger amounts of imaging and clinical data will be required. Large cross-sectional population studies based solely on volunteer participation, such as the UK Biobank15, cannot fill this gap. Even the largest current imaging studies in the field4,5, demonstrating better-than-human performance in their respective tasks, include considerably less data than, for example, ImageNet3, or the amount of data used to train algorithmic agents in the games of Go or StarCraft16,17, or autonomous vehicles18. Furthermore, such datasets often stem from relatively few institutions, geographic regions or patient demographics, and might therefore contain unquantifiable bias due to their incompleteness with respect to co-variables such as comorbidities, ethnicity, gender and so on19.

However, considering that the sum of the world’s patient databases probably contains enough data to answer many significant questions, it becomes clear that the inability to access and leverage this data poses a significant barrier to AI applications in this field.

The lack of standardized, electronic patient records is one reason. Electronic patient data management is expensive20, and hospitals in underprivileged regions might be unable to afford participation in studies requiring it, potentially perpetuating the aforementioned issues of bias and fairness. In the medical imaging field, electronic data management is the standard: Digital Imaging and Communications in Medicine (DICOM)21 is the universally adopted imaging data format, and electronic file storage is the near-global standard of care. Even where non-digital formats are still in use, the archival nature of, for instance, film radiography allows post hoc digitization, seen, for example, in the CBIS-DDSM dataset22, consisting of digitized film breast radiographs. Digital imaging data, easily shareable, permanently storable and remotely accessible in the cloud has driven the aforementioned successes of medical imaging AI.

The second issue representing a stark deterrent from multi-institutional/multi-national AI trials23 is the rigorous regulation of patient data and the requirements for its protection. Both the United States Health Insurance Portability and Accountability Act (HIPAA)24 and the European General Data Protection Regulation (GDPR)25 mandate strict rules regarding the storage and exchange of personally identifiable data and data concerning health, requiring authentication, authorization, accountability and—with GDPR—AI interpretability, sparking considerations on data handling, ownership and AI governance26,27. Ethical, moral and scientific guidelines (soft law28) also prescribe respect towards privacy—that is, the ability to retain full control and secrecy about one’s personal information. The term privacy is used in this article to encapsulate both the intention to keep data protected from unintended leakage and from deliberate disclosure attempts (that is, synonymous with ‘confidentiality’).

AI in medical imaging is a multifaceted field of patients, hospitals, research institutions, algorithm developers, diagnostic equipment vendors, industry and lawmakers. Its high complexity and resulting lack of transparency with respect to stakeholder motives and data usage patterns, alongside the facilitated data sharing enabled by electronic imaging data storage, threaten to diminish the importance of individual privacy and relax the grip on personal data in the name of, at best, scientific development and, at worst, financial interests. The field of secure and privacy-preserving AI offers techniques to help bridge the gap between personal data protection and data utilization for research and clinical routine. Here, we present an overview of current and emerging techniques for privacy preservation with a focus on their applications in medical imaging, discuss their benefits, drawbacks and technical implementations, as well as potential weaknesses and points of attack aimed at compromising privacy. We conclude with an outlook on the current and future developments in the field of medical imaging and beyond, alongside their potential implications.

Definitions and attack vectors

A glossary of the terms presented throughout the article can be found in Table 1, and a visual overview of the field can be found in Fig. 1.

Table 1 Glossary of terms encountered in the article alongside conceptual examples
Fig. 1: Secure and private AI.
figure 1

Schematic overview of the relationships and interactions between data, algorithms, actors and techniques in the field of secure and private AI.

Optimal privacy preservation requires implementations that are secure by default (synonymously privacy by design29). Such systems should require minimal or no data transfer and provide theoretical and/or technical guarantees of privacy.

The term secure AI is used for methods concerned with safeguarding algorithms, and the term privacy-preserving AI for systems allowing data processing without revealing the data itself. Their combination aims to guarantee sovereignty over the input data and the algorithms, integrity of the computational process and its results, and to offer trustworthy and transparently auditable technical implementations (structured transparency). Such systems must resist attacks against the dataset30, for example identity or membership inference/tracing31 (determining whether an individual is present in a given dataset) and feature/attribute re-derivation/re-identification30 (extraction of characteristics of an individual from within the dataset, for example by linkage attacks32). They must also withstand attacks on the algorithm or the computational process—for instance, modification of algorithm parameters (for example, by poisoning33)—or derivation of information about the dataset from them (model-inversion/reconstruction34). Finally, they must protect the data and the algorithms from theft both in storage and when transmitted over networks (asset/integrity protection).

Anonymization, pseudonymization and the risks of re-identification

Anonymization (the removal of private data from a record) and pseudonymization (replacement of sensitive entries with artificially generated ones while still allowing re-attribution using a look-up table)—collectively de-identification—are currently the most widely used privacy preservation techniques for medical datasets. In medical imaging, anonymization requires removing all pertinent DICOM metadata entries (for example, patient name, gender and so on). For pseudonymization, the true entries are replaced by synthetic data (see overview of techniques in ref. 35), and the look-up table safe-kept separately. The main benefit of both approaches is simplicity. Anonymization software is built into most clinical data archiving systems, rendering it the easiest method in practice. Pseudonymization poses additional difficulties since it requires data manipulation, not just data deletion, and safekeeping of the look-up tables for reversing the process. The latter can be problematic in the setting of insecure storage, risking data theft36. Furthermore, technical errors can render the protection ineffective and potentially (for example, in case of retaining institution names), an entire dataset identifiable. Moreover, there is substantial discourse regarding the definition of ‘sufficient/reasonable’ de-identification37 related to the objective/technical difficulty of reversing the process. Different points of view exist in different jurisdictions38, complicating the establishment of international standards. Also, de-identification techniques are usually employed as a preparation to data transfer or sharing. This presents issues in case the patient withdraws their consent, since it uncouples data governance from data ownership (impeding the right to be forgotten, GDPR article 17), or if the legislation changes. Lastly, requirements towards the de-identification process vary according to the type of imaging dataset: a radiograph of a leg is harder to link back to an individual than a computed tomography scan of their head, where the contours of the face can be reconstructed directly from the image. Such re-identification attacks39 have been shown to yield high success rates both with tabular data40,41 (such as patient records) and medical imaging data42. As a consequence, datasets more prone to identification must be processed more rigorously, for instance by removal of the face or skull region from the images (defacing/skull stripping). This complicates data handling, increasing the probability of errors and constitutes a manipulation of the imaging data, which, at worst, represents an adversarial update to the algorithm43, reducing its performance and robustness. Ultimately, even such processing might not be sufficient for the full de-identification of datasets44. Re-identified patient records are a lucrative target for health insurance companies wishing to reduce their financial risk by discriminating against individuals with certain illnesses. It has been reported that large-scale re-identification attacks and the sale of re-identified medical records have become a business model for data-mining companies45. De-identification by naive anonymization or pseudonymization alone must therefore be viewed as a technically insufficient measure against identity inference.

Decentralized data and federated machine learning

The concept of federated machine learning began gathering significant attention around the year 201546. It belongs to a class of decentralized/distributed systems that rely on the principle of remote execution—that is, distributing copies of a machine learning algorithm to the sites or devices where the data is kept (nodes), performing training iterations locally, and returning the results of the computation (for example, updated neural network weights) to a central repository to update the main algorithm. Its main benefit is the ability of the data to remain with its owner (retention of sovereignty), while still enabling the training of algorithms on the data. The federation topology is flexible (model sharing among the nodes and aggregation at a later time (peer to peer/gossip strategy47) or full decentralization, combined, for example, with contribution tracking/audit trails using blockchains48). Continuous online availability is not required since training can be performed offline and results returned later. Thus, federated learning approaches have arguably become the most widely used next-generation privacy preservation technique, both in industry49 and medical AI applications50.

While federated learning is flexible and resolves data governance and ownership issues, it does not itself guarantee security and privacy unless combined with other methods described below. A lack of encryption can allow attackers to steal personally identifiable data directly from the nodes or interfere with the communication process. This communication requirement can be burdensome for large machine learning models or data volumes. The decentralized nature of the data complicates data curation to ascertain the integrity and quality of the results. Technical research must be performed to determine the optimal method for updating the central model state (distributed optimization, federated averaging). In case the local algorithms are not encrypted, or the updates aren’t securely aggregated, data can leak or algorithms can be tampered with51, reconstructed or stolen (parameter inference), which is unacceptable from the viewpoint of intellectual property, patent restrictions or asset protection. Moreover, neural networks represent a form of memory mechanism, with compressed representations of the training data stored within their weights (unintended memorization). It is therefore possible to reconstruct parts of the training data from the algorithm weights themselves on a decentralized node52,53,54. Such model inversion or reconstruction attacks can cause catastrophic data leakage: it has been shown that images can be reconstructed with impressive accuracy and detail55, allowing visualization of the original training data. Federated learning thus offers an infrastructural approach to privacy and security, but further measures, highlighted below, are required to expand its privacy-preserving scope.

Differential privacy

Data-perturbation-based privacy approaches operate on the premise that the systematic randomized modification of a dataset or algorithm can reduce information about the single individual while retaining the capability of statistical reasoning about the dataset. The approach of retaining the global statistical distribution of a dataset while reducing individually recognizable information is termed differential privacy56 (DP). Intuitively, a dataset is differentially private if an outside observer is unable to infer whether a specific individual was used for obtaining a result from the dataset. For example, a causal relationship between obesity and cardiac disease can be inferred without knowing the body mass index of the individual patients. DP thus offers resistance to re-identification attacks such as linkage or set differencing within a certain scope of interaction with the dataset (privacy budget56). DP can be applied to the input data (local DP), the computation results (global DP) or the algorithm. Implementations range from simple random shuffling of the input data57 to the introduction of noise to the dataset (Gaussian DP58 with the benefit of better interpretability). DP can also be applied to algorithm updates during training, for instance in neural networks via differentially private stochastic gradient descent59 or private aggregation of teacher ensembles60, or during inference time. Local DP ensures privacy at the source of the data, putting the data owner in control and is thus well suited to healthcare applications61, for instance for federated learning applications in which health data are being collected by smartphones or wearable devices. DP applications to imaging are being actively explored62.

Among the challenges associated with DP, the main is the perturbation of the dataset itself. Data manipulation can degrade the data, which in an area with access to relatively little data, such as medical imaging research, may prove deleterious to algorithm performance. The technique also poses challenges with respect to plausibility testing, explaining the process to patients—that is, data legibility (human–data interaction63)—regarding algorithm development and implementation, and escalates the requirement for statistical expertise to ascertain data representativeness64. Most importantly, the specifics of implementing DP in imaging data are unclear. Tabular data can be easily shuffled, but the perturbation of images can have unpredictable effects, with research demonstrating this type of manipulation (for example, adversarial noise) both as an attack against algorithms65 and a regularization mechanism leading to increased robustness66 and resilience against inversion attacks. Thus, further research is required before the widespread application of DP in medical imaging.

Homomorphic encryption

A conceptually simple, albeit technically challenging approach to data or algorithm fortification is cryptography, widely recognized as a gold standard for information security. Current cryptographic algorithms cannot be cracked by brute force67. Encryption is easily explained to and trusted by patients and practitioners. It can be applied both to the algorithm and to the data allowing secure, joint computation.

Homomorphic encryption (HE) is an encryption scheme that allows computation on encrypted data as if it was unencrypted (plain text). Homomorphism is a mathematical concept whereby structure is preserved throughout a computation. Since only certain mathematical operations, such as addition and multiplication, are homomorphic, the application of HE to neural networks requires the operations defined within the algorithm to conform to these limitations and thus standard encryption algorithms like the advanced encryption standard (AES)68 cannot be used. Several implementations of HE algorithms69 with varying levels of efficiency exist, and the application of HE represents an efficiency–security trade-off, with computational performance currently the most notable issue. Nevertheless, HE has successfully been applied to convolutional neural networks70, and its benefits demonstrated in a ‘machine learning as a service’ scenario71, whereby data is sent over the network to be processed on an off-site server (cloud computing). It can also be used in federated learning scenarios (with or without additional DP61) to securely aggregate encrypted algorithm updates72.

Secure multi-party computation

Secure computation can be extended to multiple parties—secure multi-party computation (SMPC)73—whereby processing is performed on encrypted data shares, split among them in a way that no single party can retrieve the entire data on their own. The computation result can be announced without any party ever having seen the data itself, which can be recovered only by consensus. A conceptual example for SMPC is a ballot, where the result needs to be known, but the individual voter’s preference does not. For a technical description of SMPC, we refer to ref. 74. The research interest in SMPC has recently risen, since it allows for ‘secret sharing’ in semi-trusted and low-trust environments. Notably, SMPC has been used in the setting of genetic sequencing and diagnostics without revealing the patient’s genome75. In the domain of medical imaging, SMPC can be employed to perform analyses on datasets completely in the encrypted domain and without otherwise perturbing the data. It can thus help to increase the effective amount of available data without revealing individual identities or risking information leakage. It can also enable the ethically responsible provision of machine learning services while rendering the commercial use of the data itself impossible, or at least under the control of the individual, and subject to legal regulation after appropriate ethical debate, similar to the debate about organ donation (single-use accountability). For example, machine-learning-assisted medical image analysis services can be provided under the guarantee of data protection from malicious use in case of theft or from unwarranted financial exploitation76. As long as the data and the algorithms are encrypted, they remain unusable unless permission is granted by both parties, yielding a shared governance model. The notable limitations of SMPC are the requirements for continuous data transfer between parties (communication overhead) and for their continuous online availability. The reliability/redundancy and scalability to more than a small number of parties is a concern for SMPC applications77, and computational considerations are a concern beyond small algorithm sizes, with efficient SMPC implementations of state-of-the-art neural network algorithms currently under active development78.

Secure hardware implementations

Encryption provides a theoretical/mathematical privacy guarantee. However, privacy guarantees on the hardware level also exist, for example, in the form of secure processors or enclaves implemented in mobile devices79. They can assure data and algorithm privacy, for example, in federated learning workflows, even in the case of operating system kernel breaches. Due to the rising significance of hardware-level deep learning implementations (for example, tensor processing units80 or machine-learning-specific instruction sets81), it is likely that such system-based privacy guarantees (trusted execution environments) built into edge hardware such as mobile phones will become more prevalent.

Outlook

Medical imaging has arguably witnessed among the largest advances in AI applications due to the concurrent developments in computer vision. However, the issues of security and privacy are not limited to medical imaging82, as seen for example in the 2019/2020 SARS-CoV2-pandemic, which sparked worldwide concern about the implications of setting political, ethical and legal precedents by large-scale automatic contact tracing and movement tracking, creating a demand for their safe and privacy-protecting technical implementation83. All AI applications including sensitive data unfold in a complex, multi-stakeholder tension field of conflicting interests. The unregulated use of private data is likely to be more widespread than assumed, and cases of misuse—especially out of financial interest—will probably increase further. Yet the techniques presented here offer an opportunity to prevent stakeholder interactions from becoming a zero-sum game.

We believe that the widespread adoption of secure and private AI will require targeted multi-disciplinary research and investment in the following areas. (1) Decentralized data storage and federated learning systems, replacing the current paradigm of data sharing and centralized storage, have the greatest potential to enable privacy-preserving cross-institutional research in a breadth of biomedical disciplines in the near future84,85, with results in medical imaging50,86 and genomics87 recently demonstrated. (2) To counteract the drawbacks of the individual techniques already presented, efficient cryptographic and privacy primitives, neural network operations88 based, for example, on functional encryption89, quantization90 and optimization strategies91, and encrypted transfer learning approaches92 must be further developed. (3) The trade-offs between accuracy, interpretability, fairness, bias and privacy (privacy-utility trade-offs), need to be researched. In the field of radiology, for instance, interpretability in the encrypted setting is limited to the evaluation of trained algorithms on new images or inspection of the plain-text input data; however, intermediate outputs might be obfuscated and hard to interpret. Current research about interpretable private algorithms93 can alleviate this issue. (4) Cryptographic expertise is required for the design and implementation of secure and efficient systems that not only resist (or at least reveal) errors due to technical implementation, but are also robust against semi-honest or dishonest participants/adversaries attempting to undermine the system94. (5) Deployed models must be monitored and potentially corrected for temporal instability (that is, statistical drift95), which can be difficult with encrypted data or algorithms. (6) Until fully secure and private solutions are the standard, research has to address the question of how the right to be forgotten (for example, GDPR) can be realized—for example, via machine unlearning96 (‘un-training’ an algorithm when an individual withdraws consent). (7) The widespread implementation of secure and private AI will hinge on lowering the barrier to entry for researchers and developers by provision of accessible, open-source tools such as open-source extensions to deep learning frameworks, implementations of state-of-the-art algorithms and federated learning solutions, many of which have recently become available97,98. (8) The development of auditable and objectively trustworthy systems99 (that is, not relying on subjective assertions—for example, by governments) will promote the universal acceptance of secure and private AI solutions by individuals and policymakers. (9) The technical ability offered by secure and private AI solutions to retain sovereignty over one’s identity100 and new techniques to quantify and track the added value of individual datasets with respect to algorithm performance will strengthen the notion of private data as a scarce and valuable resource within an evolving data economy101 currently experiencing oversupply102. (10) Lastly, we view both the education of patients, physicians, researchers and policymakers, and the open scientific, public and political discourse about privacy, current risks and technical possibilities as paramount for reinforcing the cultural value of privacy and cultivating a sustainable attitude of trust and value-aligned cooperation both in science and society.