MadQCI: a heterogeneous and scalable SDN-QKD network deployed in production facilities

Abstract

Current quantum key distribution (QKD) networks focus almost exclusively on transporting secret keys at the highest possible rate. Consequently, they are built as mostly fixed, ad hoc, logically, and physically isolated infrastructures designed to avoid any penalty to the quantum channel. This architecture is neither scalable nor cost-effective and future, real-world deployments will differ considerably. The structure of the MadQCI QKD network presented here is based on disaggregated components and modern paradigms especially designed for flexibility, upgradability, and facilitating the integration of QKD in the security and telecommunications-networks ecosystem. These underlying ideas have been tested by deploying many QKD systems from several manufacturers in a real-world, multi-tenant telecommunications network, installed in production facilities and sharing the infrastructure with commercial traffic. Different technologies have been used in different links to address the variety of situations and needs that arise in real networks, exploring a wide range of possibilities. Finally, a set of realistic use cases has been implemented to demonstrate the validity and performance of the network. The testing took place during a period close to three years, where most of the nodes were continuously active.

Introduction

Quantum key distribution (QKD) technology, the ability to grow a secret key between two partners to a practically unlimited size and with bounded information leakage, has been steadily advancing since its first implementation in 1989¹. What was then a few tens of centimeters is now about a thousand kilometers^2,3 in fiber links and ground-to-satellite connections⁴. However, QKD is a demanding and still evolving technology that deals with signals at the lowest possible intensity and this imposes hard physical limits in terms of maximum absorption or tolerated noise. In the absence of quantum repeaters^5,6, point-to-point QKD links have an ultimately limited reach.

To overcome the QKD limits and serve as many users as possible in practical applications, QKD networks have been built over the last two decades⁷ and implemented all over the world to demonstrate different objectives^6,8,9,10,11. These deployments are QKD-centric, meaning that the architecture is designed to maximize the key rate while avoiding problems with the quantum channel. Additional ad hoc infrastructures, separated from the normal telecommunications network and following their own operational rules, are specifically built for this purpose. These networks, although very important and showing great advances, are also very costly and not compatible with the software and hardware architecture that supports typical telecommunications networks. None of these QKD networks are offering services to users in the same way as a standard telecommunications network does, heavily penalizing their commercialization. A tighter integration in the day-to-day telecommunications and security ecosystems, allowing for infrastructure reuse and the provision of services in a cost-effective way, is needed to grow QKD to a mainstream technology that will benefit our society.

However, the path to a QKD network that shares and integrates well with classical infrastructure, including management and operational procedures, is yet to be found. Modern and flexible networking paradigms were tested in the field¹², but more extensive research on the different technologies and their interaction is still needed.

In this paper we present a highly heterogeneous quantum network fully integrated within a commercial optical telecommunication one. It has been deployed in production networks running commercial services. It is also based on software-defined networking (SDN)¹³ since this paradigm has been demonstrated to be flexible enough to support QKD devices within a classical network architecture¹². The quantum part of the current version of the Madrid network, which we call MadQCI (Madrid Quantum Communications Infrastructure), is composed of 28 QKD modules (emitters and receivers) and a QRNG service. The devices are provided by five different manufacturers and installed in 9 production sites, separated between 1.9 and 33.1 km. Part of the network is switched, and the pairing of devices is flexible and on demand. Whenever the maximum tolerated losses allow it, direct quantum links can be dynamically established between a set of nodes while bypassing some of them. To demonstrate resilience, several links are served by multiple quantum channels using devices with different technologies and provenance. This provides an additional degree of redundancy and security also by reducing the dependence on a single manufacturer. The dynamical capacity introduced by the switching reduces the number of trusted nodes and increases the total number of possible direct quantum links to 45. Moreover, the QKD devices are in different private networks, one per manufacturer, such that they are logically disconnected to further increase resiliency and security. The optical links between nodes are just a pair of strands of optical fiber, which carry quantum and classical communications as well as service signals. In some links, several quantum channels -and the corresponding service ones- from different QKD devices using different technologies and in dissimilar configurations, shared the same fiber. To allow classical and quantum signals over the same physical infrastructure, and not to risk breaking the very strict service level agreements of the classical communications providers, a range of solutions has been deployed in the different links.

The whole network is managed and operated using the SDN paradigm, applying standards developed in the European Telecommunications Standardization Institute (ETSI)^14,15,16 for QKD. The efforts described in this paper have also helped in refining these standards, since this is the first time that they are used in such a complex network, with a physical infrastructure spanning two different network providers: Telefónica, the largest operator in Spain, and RedIMadrid, the network provider for the research and educational community in the Madrid region. Border nodes, served by two different quantum links, connect both networks and allow the creation of secure links from any node, even when these belong to different domains. Finally, a QRNG service was also integrated as a source of entropy to be used in a set of applications.

MadQCI is simultaneously operating many use cases related to different sectors. All associated classical communications needed to keep the network and use cases running, use standard equipment without any special adaptation and share the same infrastructure. This implies quantum/classical coexistence at the link level and, in most cases, sharing the same physical media. The only modification is in the encryptors used to cipher the communications. These are also commercial devices, but the firmware has been adapted to refresh the keys from the quantum network using ETSI standards. Encryption can be done at OSI network levels 1, 2, and 3 depending on the specific requirements of the use cases in terms of latencies or interfaces, so that the network services are truly transparent to the applications. This modular architecture facilitates a better integration in the security ecosystem and joint use of QKD and conventional, computational-complexity-based cryptography in a step towards crypto agility, preempting the transition of current networks towards quantum-safe ones.

The MadQCI network has been continuously operating in different stages over the last few years. Some segments have been running without interruption - except for maintenance, new software installation, etc.—for close to three years. Most of the devices have been operating over the last year, and two links have been in production during the last three months.

To the best of our knowledge, this makes MadQCI the largest and longest running QKD network in Europe. The demonstrated architecture was developed as a blueprint for future, forward looking deployments. This includes complex scenarios for exploring and demonstrating the maturity level of the technology and tackle ambitious projects such as EuroQCI, the 10 years program to build a pan-European Quantum Communications Infrastructure.

The paper first describes the logical architecture, physical devices, and optical layout of the network, delving in its integration capabilities and other significant aspects such as dynamical switching, and finally, for the sake of completeness, describes a sample of the applications that were tested, highlighting some of the specific metrics, before the concluding remarks.

Results and discussion

Architecture

Many QKD networks have already been built⁹, but they mostly share the same characteristic: they concentrate, almost exclusively, on maximizing the key throughput. To achieve this, their architecture has been tuned to minimize any disturbance in the quantum channel. Thus, the use of dark fiber for the quantum channel has been prevalent. In fact, most of these networks can be seen as ad hoc, separate networks, built solely for quantum purposes that use any classical network available for the associated classical communications. This approach requires to build a specific infrastructure just for QKD. While this might be adequate for early adopters or research-motivated but temporally limited testbeds, it presents several challenges for its widespread usage. Not reusing or not sharing existing infrastructure is very costly and demands a large investment up-front. It is not only about optical fiber, but also about additional management costs and suboptimal use of the network, dealing with proprietary interfaces, specialized maintenance, and in general, a lack of flexibility and interoperability. Such designs inhibit a scale-up of the network and adding systems in a multivendor infrastructure.

To avoid these problems, the network presented here was built following a completely different approach. Its architecture follows the SDN paradigm¹³, designed to increase the flexibility and shorten the times for deployment and maintenance. Standards and well-known tools in the telecommunications industry were extensively used to facilitate integration and adoption.

The fundamental concept of SDN is the separation between the control and data planes. In an SDN environment, the data plane, considered as the set of data and functionalities provided by the network to ensure traffic from source to destination, is bounded to dedicated elements (forwarding functions or devices). In an SDN-based QKD network, these functionalities include the key transport capability. The control and management tasks are mediated by the SDN controller, that offers a programming interface for the control of network behavior. This includes the response of the SDN-based QKD network to failures or malfunctions, e.g., detected security breaches. Moreover, the mechanisms to export the capabilities offered by the forwarding functions to the control plane are standardized. This results in a very flexible and powerful infrastructure that can incorporate new devices and technologies, facilitating interoperability and a much quicker deployment of networks and services compared to previous paradigms. At the same time, network management is also simplified since the whole network can be viewed and managed through the controller. This is what makes the SDN paradigm so popular among telecommunication companies.

From a QKD point of view, the SDN controller can obtain information on the devices installed in the network and their characteristics. QKD systems are treated as network devices that export their capabilities to the network. Note here that we are referring to the functionality, not to security-related issues, like the secret key itself that remains unknown to the controller. Depending on how much functionality the QKD module exports to the network, the integration can be as simple as commands to start, stop, and resynchronize the QKD module or as sophisticated as, e.g., to manage a single sender/receiver as an endpoint of many receivers/emitters in a one-to-many/many-to-one/many-to-many configuration. Whereas the controller does not access secure data, it knows the key requirements of different applications and can set the routes to forward keys to fulfill given service level agreements. This includes pre-emptive key storage, optical-power management in the fibers, and resource-aware optical-route planning to optimize the network for optimal performance. Dynamic optical-route planning with switches to create different sender/receiver pairs or wavelength selection is also possible as well as setting quality-of-service (QoS) parameters for specific users. Other, more sophisticated control and management tasks, such as doing network slicing, multitenancy, or creating a border node between two networks belonging to different operators, can also be performed. The functionality of creating a border node is crucial to substantially grow a network in a cost-efficient way, something that we have demonstrated in MadQCI.

The MadQCI design is shown in Fig. 1. The basic node scheme follows the approach of a software-defined QKD node (SDQKDN) that was used in a previous trial¹² and in contributions to SDN-QKD standards¹⁶.

Fig. 1: SDN-QKD architecture.

The different components (see text for details) in a node are shown within the dashed line boxes. The rounded boxes with numbers indicate the corresponding ETSI ISG QKD group specification used as an interface between the components above and below or in the connector line. We depict three nodes; each node connects with other ones in the network using a set of quantum and classical channels for quantum signal transmission, the service channels to keep the quantum links working, the key distillation, and user data channels. Other classical channels can be established between the encryptors or software applications running in the node. Not all nodes serve user applications since they can be used as trusted relays with the only purpose of extending the QKD reach. For this reason, relay nodes have no KMS functionality. Key routing, which cannot be considered a KMS functionality, is managed by the Forwarding Module. The middle node is an example of a relay node. We encapsulate all the functionality needed to create the secure key transport capability that characterizes a QKD network in the lowest layer, the quantum forwarding plane, analogous to the data forwarding plane in standard SDN networks. It includes the QKD modules together with a forwarding module that implements a secure key forwarding mechanism.

The network nodes can have several QKD modules installed. They interact through interfaces with a set of disaggregated software components implementing clearly defined functions¹⁷. This allows for the design of interfaces in a vendor-independent manner for structured communications. The interfaces were implemented with well-known tools and adhering to standards in telecommunications, which also helps to create confidence in the technology. The approach is scalable and flexible: it can be extended to many nodes, be used to increase the capacity of each node, support a variety of QKD technologies, and even support new services beyond QKD. The components within a node are:

• Local key management system (LKMS): it collects the keys from the QKD modules and serves the applications; indexes and stores the generated keys, manages their lifecycle, and keeps track of the key-generation peers; provides information on key availability to the SDN controller through the SDN agent and the keys to be forwarded when needed. Below we use the general term Key Management System (KMS) to address the functionalities of the set of all LKMSs.

• Forwarding module: it is in charge of the key transport between nodes using the shared keys created by the QKD module pairs. In contrast to typical implementations, this functionality is here separated from the LKMS, since key routing is not a part of KMS duties as defined in, e.g., the NIST SP 800 document series. This facilitates the integration into the standard security ecosystem.

• SDN Agent: SDN controller counterpart in each node that connects the controller with all the components within the node. Note that security-sensitive information is not available to the control mechanism.

• QKD Module: the quantum sender/receiver itself, which continuously generates the keys. In general, there are three channels associated with it: the quantum channel, a service channel needed to stabilize the quantum channel (possibly integrated with the former), and the classical key-distillation channel.

• Application: any entity, inside the SDQKDN security perimeter, requesting QKD keys from the LKMS. The applications might be external, e.g., an end-user application, a hardware security module (HSM), a virtual network function, or internal with respect to the key distribution functionality, e.g., authentication, virtual link management, or key transport. The applications use the application interface implemented in the LKMS to obtain the key material.

This set of components is sufficient to implement all the functionality required by a QKD network. They are also flexible enough to cope with new applications and substantial enough to represent a possible target for standardization. Following the SDN paradigm, the node communicates with the (logically) centralized SDN controller that implements all necessary logic and interfaces to control the network. The controller creates the logical and physical connections necessary for sharing a key end-to-end. It also provides the interface to the network management system, allowing advanced functionalities, e.g., setting QoS parameters for different users, the orchestration of several networks, and the creation of large multi-domain, multi-tenant networks. A sought-after effect of this disaggregated approach, open-standard interfaces, and communications is to allow for vendor independence and to reuse as much of the existing communications technology as possible. This is done again with the objective of creating confidence and allowing an as easy as possible integration of QKD technology in the communications and security ecosystem.

The nodes of MadQCI were deployed in the network as shown in Fig. 2. This network was not created ad hoc but uses a pre-existing production network that provides services to commercial customers. It is important to remark that all the installed QKD systems were located in production facilities under typical, carrier-grade, working conditions. No system was operated in a lab environment except the experimental link that was connected remotely for a limited time. In total, 28 QKD modules (counting emitters and receivers, 26 on-site and 2 remote) using different QKD technologies and protocols were installed (see Table 1) in 9 nodes of the Telefónica and RedIMadrid production networks. Both networks were connected through special border nodes. The length of the links ranged between 1.9 and 33 km (optical losses between 2.0 and 14.3 dB in the C-band), covering the Madrid metropolitan and suburban area. Except for one link (link 3 in Fig. 2) that uses two pairs, all nodes are connected through a single pair of optical fibers that carry all quantum and classical signals. No ad hoc fiber was deployed specifically for the quantum channel. The coexistence of the quantum and classical channels was a must, as well as compatibility with standard optical transport (OTN) equipment and cryptographic appliances. For maximum flexibility and transparency to the application layer, encryption can be done at OSI levels 1, 2, and 3. Level-3 encryption (IPsec) was done via software implementation of AES, as well as one-time pad encryption. Off-the-shelf OTN and network encryptors from ADVA and Rohde & Schwarz were used. The R&S firmware of the level-2 encryptor was adapted to extract QKD keys with the ETSI GS QKD 004¹⁴ standard and use them to cypher communication using AES at rates up to 40 Gbps. Encryption at level-1 was done using the ADVA encryptors, also modified to accept the key using the GS QKD 014¹⁵ standard.

Fig. 2: Physical layout of Madrid Quantum Communications Infrastructure.

It is composed of 9 sites in two production networks with 26 QKD modules on-site and 2 experimental ones installed remotely (not depicted). White labels refer to information about the links. Gray labels refer to nodes. The links in red belong to RedIMadrid, the network provider for all research and educational institutions in Madrid. The ones in blue belong to Telefónica. A border link (yellow) connects the two networks. All nodes are in production facilities, sharing the location with classical equipment and optical fibers carrying classical communications. Each link is marked with a label (number in the rounded part of a white label) for further reference, together with the optical losses, physical distance, and number of quantum and classical channels on the same fiber. Each node is tagged with the number and type of QKD modules installed, the OSI level of the classical hardware security module (Level-3 and OTP were always available through software), and whether there is an optical switch for the quantum channel installed. The switches allow the creation of direct quantum channels among non-contiguous nodes, raising the total number of possible connections to 45, up from the nine possible ones in the physical topology depicted above (see Table 1 for additional details).

Table 1 QKD systems installed in each link and their main characteristics

An important aspect of the network is that the optical connection infrastructure is not static. Several all-optical switches, managed through the SDN controller, were installed. Specifically, the quantum channels were not static and could be established with different endpoints. Again, we did this using standard telecommunications technology, both at the hardware and software level (OADM modules built from standard, readily available components, and Transport API) to demonstrate compatibility. In this way, we had many more direct connections (i.e., with an uninterrupted quantum channel) than those strictly linking one node to its nearest neighbor. A total of 45 compatible direct connections were possible with optical losses low enough to create QKD keys, which is substantially more than if it was a traditional, fixed QKD network physically laid out as in Fig. 2. The capability of managing switches can be used to seamlessly include MDI type QKD systems, since the central measurement station can be seen, from a topological point of view, like a switch¹⁸. The SDN controller, together with the LKMS and key forwarding module, can distribute end-to-end keys between any two nodes in the network, no matter the vendors or combination of these in the connecting path. The controller can also regulate the key provisioning, thus supporting QoS constraints and making the network more resilient to connectivity failures.

QKD systems

To show that many QKD devices, not just in quantity but also in type, can interoperate in a network during considerable periods of time is a key requirement for operators before deploying the technology in the real world. This heterogeneity was specifically sought after when designing the network. In this section we describe the deployed QKD technology.

The 10 Huawei CV modules were continuously available, except for servicing, to run the use cases close to three years. The 8 ID Quantique DV modules were running most of the time, in periods of months, during the last two years, while 4 Toshiba DV modules were installed and running continuously during the last year and other 4 additional ones during the last three months, when the network was running in its full configuration. The two, experimental, QKD modules from AIT were connected, albeit remotely, to the network during shorter time periods to demonstrate how to adapt new devices easily. However, no use cases were run using them. The distance between emitter and receiver in the remote link was 4.3 km. To further show the flexibility of the approach, a QRNG provided by QuSide was integrated in the network. The service could be used to produce on-demand, high-quality random numbers from an independent vendor for, e.g., key-generation purposes. A detailed list of the modules and their main characteristics is collected in Table 1.

AIT CV-QKD modules

The QKD modules developed by AIT provide a fully integrated CV system housed in two 19” inserts (Alice & Bob). It uses a QPSK constellation with 100 MBaud symbol rate for the quantum states and a polarization multiplexed pilot tone as a phase reference. All digital driving signals are generated on an FPGA platform in the transmitter device, ready for receiving random numbers from a physical QRNG device. Auxiliary signals for packet triggering and clock synchronization are wavelength-multiplexed directly on the quantum channel, enabling a true single-fiber operation. In the receiver, a true local oscillator is employed for heterodyne coherent detection, together with a 90° optical hybrid and two balanced receivers each for the quantum signals and the pilot. Automatic feedback loops for polarization control and laser frequency stabilization are implemented to ensure a stable long-term operation. Separate computers on either side perform the system hardware control, digital signal pre-processing, and real-time post-processing. Pre-processing consists of down-sampling, frequency offset correction, phase correction, and parameter estimation. Parameter estimation is required for several algorithms in the post-processing stack, including calibration measurements of the thermal system noise and optical shot noise, an estimation of the SNR of the quantum states as well as their excess noise. The post-processing pipeline is instantiated on either side and communicates on an authenticated classical channel. The first step is an optional post-selection algorithm, which may improve the overall performance by trading the raw key rate for a better SNR. Next is information reconciliation by means of LDPC error correction followed by a confirmation of its success. Finally, all information leaked to a potential eavesdropper during physical key exchange and classical post-processing is rendered useless by performing privacy amplification. Here, an upper bound of the leaked information is calculated following the theoretical security proof. The final key is then reduced by this amount, and all classically disclosed information (e.g., during information reconciliation) in a hashing algorithm forms the secure key. In the demonstration within MadQCI, whose purpose was to show the capability to easily integrate new, even experimental devices, in a running QKD network, those keys were forwarded to a KMS remotely over a raw TCP connection.

Huawei technologies Duesseldorf GmbH (HWDU), CV-QKD prototypes

As a partner in CiViQ and in the OpenQKD open calls, HWDU supplied 10 CV-QKD modules (5 senders and 5 receivers) to MadQCI. These modules are flexible, any sender could interoperate with any receiver, whereby the quantum channel between them could be optically switched, and moreover, as the lasers in both the senders and the receivers are tunable, wavelength switching in a broad range was feasible. Considering different possible paths between the 7 module locations, 36 different QKD links were supported, far more than the five links that would be available in a static configuration.

Each pair of the low-noise and low-complexity modules is generating key by means of a Gaussian-like modulation of coherent states and heterodyne coherent detection. It is to be noted that we are approximating true Gaussian modulation, an approach that is known to yield insignificant differences in key rates as a function of attenuation, compared to the theoretical case of analytic modulation¹⁹. The symbol rate is 12.5 MBd. Both, sender and receiver modules, feature phase and polarization diversity. Further, in-band synchronization is supported, whereby only one dense wavelength division multiplexing (DWDM) channel in the C-band in one direction is needed for the QKD operation. Additionally, a bidirectional, standard internet protocol (IP)-based post-processing link, that can operate on any existing network infrastructure, is required. The sender modules can transmit any chosen value between 0.0004 and 40 photons per symbol on average in the quantum band. Optical samples are generated, detected, and transformed from and to symbols utilizing appropriate DSP algorithms, running predominantly on an FPGA-based SoC. Post-processing of symbols follows the traditional steps for CV-QKD, whereby error correction runs with a single fixed-rate code, which supports a signal-to-noise ratio (SNR) down to −19.5 dB. This is a receiver sensitivity of ~−105dBm with 2.5 dB of receiver loss and heterodyne detection. Excess noise powers smaller than 50 dB below the shot noise can be detected. With trusted detector noise and inherent system noise as low as 0.15 mSNU receiver side, a single sender-receiver pair supports up to 23 dB of channel loss.

ID Quantique’s Cerberis 3

In the framework of the OpenQKD project, ID Quantique supplied in total 16 pairs of QKD modules to different testbeds across Europe supporting different use-cases during time spans ranging from few months to more than one year. MadQCI received four fixed pairs connecting each transmitter to its corresponding receiver. Two links have the quantum channel at the O-band (1310 nm) and other two in the C-band, at ITU channels 32 and 34. The 1310 nm systems were delivered with additional built-in spectral filters to allow multiplexing of the quantum channel with classical channels in the C-band. The systems are Cerberis 3 products, which were the predecessor of the current product Cerberis XG.

The implemented QKD protocol of the Cerberis 3 is the Coherent One Way (COW) with time-bin encoding²⁰. COW does not require phase randomization between consecutive pulses and uses a simpler decoy mechanism. This reduces the complexity of the optics for preparation and detection. The MadQCI devices implemented the last version of the SW, which include the countermeasure to the recently discovered attacks on the COW protocol. This countermeasure has little impact on the performance of the system. However, the typical real-world conditions, including temperature changes depending on time and local environments, caused larger than expected fluctuations in its performance. The new generation of QKD products is designed to deal with non-optimized cooling situations.The production environment of MadQCI enabled the optimization of stable operation.

The Cerberis 3 product includes a complete software suite to build up stand-alone trusted node QKD networks composed of ID Quantique devices. The network control and management are then run on an external server. However, to demonstrate and test the MadQCI interoperable setup, the Cerberis 3 were integrated as simple point-to-point links, and keys were requested via ETSI GS QKD 014 by the local KMS module and managed by the global, SDN-aware KMS, enabling network-wide end-to-end secure key transport. This demonstrates the flexibility which operators have already today to build up complex QKD networks.

Toshiba QKD devices

The four QKD systems (eight modules in total) from Toshiba implement the T12 protocol²¹ which is an optimized version of BB84 with weak coherent pulses and decoy states. Phase-encoded quantum states at 1 GHz clock rate are generated by a gain-switched laser followed by an asymmetric Mach-Zehnder interferometer in the QKD transmitter²². In the receiver, self-differencing avalanche photodiodes (APDs) are used to measure the received states. The system fits into a standard 19-inch data center rack and occupies 3 rack units (3U) per node.

The Toshiba QKD devices in MadQCI all used a quantum channel wavelength of 1310 nm, optimized for supporting co-propagation of quantum and classical signals by maximizing the spectral separation between multiplexed quantum and classical channels. In addition, the systems include high-extinction narrow-band spectral filtering and time-domain gating of the APDs to further isolate the quantum channel from co-propagating classical/Raman-scattered light with maximal signal-to-noise ratio. This enables, for example, co-propagation of quantum light with over 60 × 100 Gbps DWDM channels in the C-band over 50 km, while still maintaining >100 kbps QKD secure key-generation rates.

The QKD systems also include automatic self-optimization routines to dynamically adjust various optical parameters (e.g., polarization, phase, timing delays etc.) to maintain long-term maximal performance on each communication link²². Add/drop multiplexing hardware is also included in the QKD unit to multiplex the quantum channel, QKD service channels and any auxiliary data channels onto the communication link. Finally, QKD-generated keys are exported to the MadQCI network KMS at each node using the standardized ETSI GS QKD 014 interface.

Quside QRNG

As an additional service in the network to be used in the MadQCI use cases, a Quantum Random Number Generator was made available. The device consists of a quantum entropy source (QES) based on the proprietary phase diffusion technology²³ together with the firmware required to calibrate, control, monitor, and provision the entropy generated in the QES to an operating system. It is able to produce very high-quality random bits at a speed of up to 4 Gbps. The QES and control electronics are integrated into a commercial PCIe card that was installed in a regular server located in the Telefónica network. Subsequently the QKD systems were used to transport the key from the QRNG through the network using one-time-pad encryption for the use cases where it was needed. The PCIe card is attached to a virtual machine where the corresponding drivers and libraries are installed and a server leveraging the libraries was deployed to provide entropy on demand through a simple REST interface. The key relay service present in the network requested entropy from this server for encryption key generation.

Optical transport and encryptors

As mentioned above, encryptors modified to fetch keys from the local key management systems for the encryption were used. For the R&S SITLine ETH layer 2 Ethernet encryptors it was important that they could continue to operate in their view of the classical (possibly meshed) layer 2 network. The abstraction of the ETSI QKD key application interface (GS QKD 004) allows to get keys between any logical pair of devices which can and want to communicate without configuring the specific QKD node topology into the encryptors. Then the keys could be used in a hybrid way, combining the QKD key with a classical key exchange mechanism and using the already existing integrated classical key management functions of the devices to keep all existing network functionality of the layer 2 encryptors. It was also important to keep the existing approved classical key management to minimize the cryptographic-relevant changes, ease a later approval of the QKD enhancement, and have a fallback to classical security in case of problems with the QKD modules. The payload data is then AES-256 encrypted on one or up to four 10 GbE interfaces.

Optical layout

The devices described in the last section were deployed in production facilities in Madrid (Fig. 2). Typically, several QKD systems were installed in each node and the low-level optical structure for the quantum and associated devices of the network is outlined in Fig. 3. Since it was a main target to explore how to include quantum communications in a standard production network, one of the guiding principles was to limit as much as possible the changes to the underlying standard optical network configuration. In this spirit, modifications were limited to add the mandatory multiplexers to add/drop the corresponding wavelengths and the already mentioned SDN-controllable optical switches to create additional quantum channels. In particular, the power of the classical channels was left untouched; it was not modified in order to reduce the noise in the quantum channels. The global control of the network was done through the SDN controller described above. In Fig. 3, the connections between the classical and quantum part of the network are located at the input/output ports of the multiplexers at the exit/entry points of each node. These ports are typically marked with the rate of the classical channels entering/exiting the fiber at that point (e.g., #x10GE, meaning that in that multiplexer a number of # 10 Gigabit Ethernet classical channels enters/exits the multiplexer).

Fig. 3: Low-level optical structure of the quantum part of MadQCI testbed network.

It shows the optical setup of the network. It includes the quantum and directly associated equipment. The multiplexers and number of fibers are only shown as indicative due to complexity reasons. The direct links follow the same numbering as in Fig. 2 and are numbered L1 through L9. (Li meaning link “i”, with “i” the link number in the white labels in Fig. 2) The three rightmost nodes form a separate domain and are operated by Telefónica containing Norte, Concepción, and Distrito. The Telefonica Norte site is connected to the Quevedo site in the RedIMadrid network over link 4 forming the border link. It is served by two QKD systems from two different manufacturers operating in different CV and DV modes. The longest link is link 8 between Quijano and Quevedo with a length of 33 km. The classical part of the network connects with the quantum part at the input/output ports of the multiplexers at the entry/exit points of each node. They are marked in blue and labeled as #10GE, which means, for example, that a number of # 10 Gigabit Ethernet classical channels go through that multiplexer. Because of the density of the figure, specifically in the Telefónica domain, not all possible classical channels are indicated. In the RedIMadrid domain, most of them are made explicit.

In any case, safety measures were taken to guarantee that the classical links were never disrupted. Backed by industrial-grade solutions, these links could not be harmed by any of the tests in order to fulfill the strict service level agreements. In the end, after almost three years of testing many different use cases and configurations, it is interesting to highlight the performance, stability, and classical compatibility of these links since it clearly shows how QKD technology has matured over time and can be used jointly with classical communications.

Physically, the network has two different domains. The three rightmost nodes shown in Fig. 3 are the Telefónica domain. This domain is connected with the RedIMadrid domain, with all the remaining nodes, through the border link connection (link 4: Quevedo-Norte). Since it is important for network operators to ensure that they can procure the equipment from different manufacturers and test their compatibility at the physical and logical level, the mixture of devices is larger in the Telefónica domain. In this domain all three manufacturers were present and the same physical media, a pair of optical fibers, was shared between the classical and QKD links.

The border link is served by two QKD pairs, one is DV (ID Quantique) and the other CV (HWDU). This critical link uses two technologically different QKD systems in the spirit of enhancing resilience and security by using a mixed configuration, where the final secret keys are obtained from two different sources.

The optical spectrum in one of the shared fibers in this link (direction Quevedo→Norte) is shown in Fig. 4 (left side). It presents the C-band spectrum (from 192 to 194 THz, 1546 to 1562 nm, approximately). The notch where the CV quantum channel is placed (within Ch37/Ch 38) can be seen together with other three 10 G telecom data channels (Ch21-Ch23) and the service channel for the DV-QKD system (Ch30). In this link, the DV quantum channel is located in the O-band. It is interesting to mention that moving to 100 G data channels reduces the noise in the quantum channel, since the higher speed connections are more bandwidth efficient, and the optical power leaked out of its nominal wavelength is lower.

Fig. 4: Optical spectra of selected links.

The border link number 4 (a) and the longest link 8 (b) are displayed. The OSA capture of link 4 in the C-band in the direction Quevedo→Norte shows three data channels (Ch21-Ch23) as well as the service channel of the ID Quantique system at Ch30 co-propagating with the quantum signals. Whereas the quantum channel of the ID Quantique system is in the O-band and therefore outside of the measurement range, the quantum channel of the HWDU device co-propagates in this fiber at the C-band (Ch 38 or 37, depending on the switched path) where a notch can be seen. On the right side, the C-band spectrum of link 8 (33.1 km and 11.8 dB losses) is shown. In this link a DV quantum channel in the O-band co-propagates with the classical data channels (25-29-34), the service and distillation channels (59–60) in the C-band.

Figure 4 (right) shows the optical spectrum of link 8, the longest direct link in the network (331 km) that is served by a DV-QKD system provided by Toshiba. Similar to the QKD system of link 4, the quantum channel is in the O-band while all the data, including encrypted traffic, and service channels are in the C-Band. This reduces the noise in the quantum channel at the expense of higher losses (~50% more in the O-band compared to the C-band). In the measured optical spectrum again only telecom channels are visible, although they include the service and key-distillation channels serving the quantum link.

To give a more detailed view on the performance in different links, Fig. 5 shows the QBER and secret key rate graphs that correspond to those discussed in Fig. 4 (Links 4 and 8 in Figs. 2 or 3). Link 8 (Fig. 5, right side) is the longest link in the network and is served by a DV (Toshiba) system. The quantum channel is in the O-band while all the data, including encrypted traffic, and service channels are in the C-Band. This reduces the noise in the quantum channel at the expense of ~50% higher losses. Note that the performance in both links varies significantly; although both are DV, the QKD protocol and basic parameters of the systems are quite different. One uses a COW protocol while the other is an optimized BB84 decoy states protocol. Note that the scale used for the key rate differs by an order of magnitude between both sides. Besides this, it was found that the Cerberis 3 generation of systems were very sensitive to variations in temperature, even those inside a datacenter, producing a secret key rate with more fluctuations than expected. QBER and secret key rate in these links are displayed for a period of about half a year, highlighting the long-term stability of the network. The performance, stability, and classical compatibility of these links clearly shows how QKD technology has matured overtime.

Fig. 5: Examples of measured QKD performances.

a shows the link 4 (border link). The QBER and secret key rate over a period of about half a year is shown. Segments without data were due to maintenance labors. b shows link 8 (longest link) QBER, and secret key rates of the QKD system provided by Toshiba are shown for a similar period. The disruptions seen are due to maintenance work.

In principle, quantum and classical signal coexistence, sharing the available optical fiber, was achieved in different ways in different links (see Table 1 for the detail of what is installed in each link): CV-QKD systems are intrinsically more resilient to noise. The CV detection acts as electronic filtering that is much narrower than optical filtering, which allows to use the C-band for both, quantum and classical channels. DV-QKD systems have been usually operated with the quantum channel at the O-band, while keeping the classical channels in the C-band. This allows to avoid the Raman scattering from the classical channels, which is very difficult to filter because it is produced over a broad spectrum. The large spectral separation between the O and C bands, ensures that the Raman-scattered light in the O band because of the classical channels in the C-band, is very little, reducing the chance to have a scattered photon in the same gate as the quantum signal. The direction of the quantum channel coincided with the direction of the telecom operation. The integration of DV-QKD devices operating in the O-band was not evaluated systematically and independently by network operators so far. Within MadQCI, a comparison of different operation scenarios was possible.

Additionally, other configurations were also tested to address less standard situations. In Link 2 (Fig. 3), a DV-QKD link and CV-QKD link were operated together in the C-band on a single fiber for the quantum channels, while the second fiber in the pair was used for all data telecommunication channels. For the latter, bidirectional transceivers and multiplexers were used creating the upstream and downstream classical channels over the second fiber. This configuration is not very common, although it is supported by manufacturers. Interference among quantum channels was easily prevented by simple measures like avoiding the use of exactly the same wavelengths.

The logical compatibility and transparency towards the applications operated by users even from different domains was guaranteed by the software stack built for the network following the SDN mechanisms described above. At large, MadQCI has the focus of heterogeneous integration without restrictions to deploy hardware from different suppliers operated in different domains, and transparency to the applications, that do not need to worry about the complexities of the underlying network. This was demonstrated through the implementation and testing of many different use cases.

Use cases

To underline the maturity of QKD technology and assess its feasibility, the network was running many qualitatively different use cases during the last three years. A short description of the most significant ones is provided in the supplementary material. A detailed description of the results is beyond the scope of this contribution and will be published elsewhere. This section is included here just to provide a general idea of the global performance from the network and applications point of view and the type of results obtained, given the broad range of applications tested.

Since it is the practical network usability what concerns us here, the traditional device-centric figures of merit presented in Fig. 1 (SKR, QBER, etc.) have a limited significance. With the architecture and redundancies available it is clear that there is enough key rate for various security applications that consume key, but that does not immediately translate in key indicators to judge the performance of an application, and other telecommunications metrics closer to the use case, are needed.

The use cases studied cover a wide range of services, from critical infrastructure protection, secure network management, cloud, 5 G, and final user services (e.g., e-health), as well as experimenting with new protocols, not directly related to encryption. The corresponding high-level application metrics can be very different, not only in performance numbers but also in significant magnitudes. In some use cases (5 G or those related to real-time applications), latencies are the key figure, while for others, it could be the throughput, the maximum number of applications being served simultaneously, etc.

More than 85,000 use cases instances have been executed to gather these high-level, application-specific and network data to derive the metrics relevant to the particular cases. A detailed discussion of the application topologies would be very exhaustive. However, to illustrate the variety of this type of metrics, not commonly seen in research papers concentrating on the low-level performance of single links, but highly relevant for the network design and operation and for the final user, we show several of them as examples in Table 2.

Table 2 Metrics for different use-cases and sample values

To be more precise in the illustration of the differences between the low-level QKD metrics and the application-level ones, we present a little more detail for a selected use case depicting the latencies in the Ordered Proof of Transit (OPoT)²⁴ use case, see Fig. 6. A detailed analysis of all the use cases is out of the scope of the present paper.

Fig. 6: Network metrics for the OPoT (Ordered Proof of Transit) use case (see text).

In this case, the relevant metric is the latency. Each network packet is processed and tagged using a method based on QKD¹⁷. This serves the purpose of guaranteeing that the packets have passed through a specific set of nodes and in a specified order. Since the number of packets is very large, it is important to make sure that the increased latency penalty is limited. The plot shows that the average latency increase is ~5.86 ms for packets with OPoT (red), to be compared with the 3.26 ms without OPoT (green). The measures were taken during a period of a few days to demonstrate the stability of the metric during a long working period. The peaks typically correspond to times in which there was no key available due to the interaction with other use cases running simultaneously.

OPoT is a networking application that targets the problems of network security and attestation. The problem is to make sure that the data packets have passed through a defined set of nodes (e.g., a firewall) and in the correct order. The OPoT method used here requires QKD keys to prove this, and, since the number of packets in the network can be huge, the processing time must be as low as possible. Hence, latency is a key figure of merit. The latencies with and without OPoT are presented in Fig. 4, showing that the approach is, indeed, feasible within a modern-day QKD network as presented here.

In the Madrid quantum network, in contrast to past efforts where the quantum part was a specially built, ad hoc network, we have developed a network to demonstrate the integration of QKD within production networking and security ecosystems. We believe that this integration, facilitating access to quantum communications as an easy-to-use, scalable service, and where much infrastructure can be shared to avoid large up-front costs, is key to the development and broad adoption of QKD and, in general, quantum communications.

To this end, we have used as a deployment base two already running, production-grade networks, with very little or no modification of their physical infrastructure to add quantum communications. The effort was put into the logic that glues the quantum and classical networks and enables them to share much of the infrastructure. To achieve this, we have used the software defined networking paradigm. This is a widespread paradigm in classical telecommunications networks that achieves its flexibility by decoupling data forwarding from control and management planes. QKD devices can become part of a specific forwarding plane, exporting their capabilities and requirements to a logically centralized SDN controller, where we have built the logic to control and enable quantum communications integrated within the classical network. In this way, we can seamlessly add quantum capabilities to an existing network. This was demonstrated using approved standards and very heterogeneous QKD, optical transport, and security hardware, together with a range of software tools and protocols well-known by telecommunications companies. In this way, showing manufacturer independence, standardization, and common toolsets, we expect to create confidence and facilitate the widespread adoption of quantum communications technologies.

The capabilities of the network were demonstrated on a large set of nodes, many with several QKD modules installed and deployed using this new architecture. A typical security infrastructure based on standard hardware encryptors, and protocols was also implemented together with the network. QKD keys were also mixed with conventional keys from standard public-key cryptosystems, to achieve a better integration with the security ecosystem. A large set of use cases was run for testing and performance purposes, gathering low-level as well as application metrics. They were quite diverse, ranging from applications to secure critical infrastructures or network control to cloud and 5 G applications. The testing took place over a period of close to three years, where most of the nodes were continuously active. The results clearly show the feasibility of this approach to building large QKD networks.

MadQCI has brought together the highest number of different QKD links operated in a complex production network in Europe and during the largest period so far, showing that they can be integrated into the telecommunications and security infrastructures using modern network paradigms, tools, and standards and that this can be done using existing, production-grade telecommunications networks, as a base. The network has demonstrated interoperability among manufacturers and network operators and potential for scalability that can act as a blueprint with strong implications for a future European-wide QKD network infrastructure.