Abstract
The rapid growth in the amount of data being transferred within data centres, combined with the slowdown in Moore’s Law, creates challenges for the future scalability of electronically switched data-centre networks. Optical switches could offer a future-proof alternative, and photonic integration platforms have been demonstrated with nanosecond-scale optical switching times. End-to-end switching time is, however, currently limited by the clock and data recovery time, which typically takes microseconds, removing the benefits of nanosecond optical switching. Here we show that a clock phase caching technique can provide clock and data recovery times of under 625 ps (16 symbols at 25.6 Gb s−1). Our approach uses the measurement and storage of clock phase values in a synchronized network to simplify clock and data recovery versus conventional asynchronous approaches. We demonstrate the capabilities of our technique using a real-time prototype with commercial transceivers and validate its resilience against temperature variation and clock jitter.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
Source data are provided with this paper.
References
Singh, A. et al. Jupiter rising: a decade of Clos topologies and centralized control in Google’s datacenter network. ACM SIGCOMM Comput. Commun. Rev. 45, 183–197 (2015).
Markov, I. Limits on fundamental limits to computation. Nature 512, 147–154 (2014).
Dorren, H. et al. Challenges for optically enabled high-radix switches for data center networks. J. Lightwave Technol. 33, 1117–1125 (2015).
Ghiasi, A. Large data centers interconnect bottlenecks. Opt. Express 23, 2085–2090 (2015).
Krishnamoorthy, A. et al. From chip to cloud: optical interconnects in engineered systems. J. Lightwave Technol. 35, 3103–3115 (2017).
Testa, F. & Pavesi, L. Optical Switching in Next Generation Data Centers (Springer, 2017).
Ballani, H. et al. Bridging the last mile for optical switching in data centers. In 2018 Optical Fiber Communication Conference (OFC) W1C.3 (OSA, 2018).
Clark, K. et al. Sub-nanosecond clock and data recovery in an optically-switched data centre network. In 2018 European Conference on Optical Communication (ECOC) 1–3 (IEEE, 2018).
Chen, C. P. et al. Programmable dynamically-controlled silicon photonic switch fabric. J. Lightwave Technol. 34, 2952–2958 (2016).
Cheng, Q., Wonfor, A., Wei, J. L., Penty, R. V. & White, I. H. Low-energy, high-performance lossless 8 × 8 SOA switch. In Proc. 2015 Optical Fiber Communication Conference (OFC) Th4E.6 (OSA, 2015).
Shi, K. et al. System demonstration of nanosecond wavelength switching with burst-mode PAM4 transceiver. In Proc. 2019 European Conference on Optical Communication (ECOC) 1–3 (IEEE, 2019).
Xilinx. DS893 (v1.12) Virtex UltraScale Architecture Data Sheet: DC and AC Switching Characteristics (2019); https://www.xilinx.com/support/documentation/data_sheets/ds893-virtex-ultrascale-data-sheet.pdf
Banu, M. & Dunlop, A. E. Clock recovery circuits with instantaneous locking. Electron. Lett. 28, 2127–2130 (1992).
Terada, J. et al. A 10.3 Gb/s burst-mode CDR using a ΔΣ DAC. IEEE J. Solid-State Circuits 43, 2921–2928 (2008).
Rylyakov, A. et al. A 25 Gb/s burst-mode receiver for low latency photonic switch networks. IEEE J. Solid-State Circuits 50, 3120–3132 (2015).
Yang, Y., Wen, Y. J., Nirmalathas, A., Liu, H. F. & Novak, D. Optical clock recovery at line rates via injection locking of a long cavity Fabry–Pérot laser diode. IEEE Photon. Technol. Lett. 16, 1561–1563 (2004).
Ozkaya, I. et al. A 56 Gb/s burst-mode NRZ optical receiver with 6.8 ns power-on and CDR-lock time for adaptive optical links in 14 nm FinFET CMOS. In Proc. 2018 IEEE International Solid-State Circuits Conference (ISSCC) 266–268 (IEEE, 2018).
Bostica, B., Burzio, M., Gambini, P. & Zucchelli, L. in Photonic Networks (ed. Prati, G.) 362–376 (Springer, 1997).
Timing and Synchronization Aspects in Packet Networks G.8261 (ITU, 2008); https://www.itu.int/rec/T-REC-G.8261-201908-I/en
Lee, K. S., Wang, H., Shrivastav, V. & Weatherspoon, H. Globally synchronized time via datacenter networks. In Proc. 2016 ACM SIGCOMM Conference 454–467 (ACM, 2016).
Slavík, R. et al. Ultralow thermal sensitivity of phase and propagation delay in hollow core optical fibres. Sci. Rep. 5, 15447 (2015).
Data Center Networking Equipment—Issues and Best Practices TC9.9 (ASHRAE, 2016); https://tc0909.ashraetcs.org/documents/ASHRAE_TC0909_Power_White_Paper_22_June_2016_REVISED.pdf
Soleimani, S., Afzali-Kusha, A. & Forouzandeh, B. Temperature dependence of propagation delay characteristic in FinFET circuits. In Proc. Int. Conference on Microelectronics (ICM) 276–279 (IEEE, 2008).
Ma, L.-S., Jungner, P., Ye, J. & Hall, J. L. Delivering the same optical frequency at two places: accurate cancellation of phase noise introduced by an optical fiber or other time-varying path. Opt. Lett. 19, 1777–1779 (1994).
Lord, A., Blank, L. C., Boggis, J. M., Bryant, E. & Stallard, W. A. Theory of control mechanism for an optically time-division-multiplexed system. Electron. Lett. 24, 2011–2012 (1988).
Ellis, A. D., Widdowson, T., Phillips, I. D. & Pender, W. A. High speed OTDM networks employing electro-optic modulators. Trans. Inst. Electron. Inf. Commun. Eng. Sect. E E81-C, 1301–1308 (1998).
Zhang, Q., Liu, V., Zeng, H. & Krishnamurthy, A. High-resolution measurement of data center microbursts. In Proc. 2017 Internet Measurement Conference (IMC) 78–85 (ACM, 2017).
IEEE Standard for Ethernet—Amendment 3: Physical Layer Specifications and Management Parameters for 40 Gb/s and 100 Gb/s Operation over Fiber Optic Cables 802.3bm (IEEE, 2015); https://standards.ieee.org/standard/802_3-2018.html
Lipínski, M., Włostowski, T., Serrano, J. & Alvarez, P. White rabbit: a PTP application for robust sub-nanosecond synchronization. In Proc. 2011 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control and Communication (IEEE, 2011); https://doi.org/10.1109/ISPCS.2011.6070148
Bousonville, M. et al. New phase stable optical fiber. In Proc. Beam Instrumentation Workshop 2012 (BIW2012) 101–103 (JACoW, 2012).
40-Gigabit-Capable Passive Optical Networks 2 (NG-PON2) G.989.2 (ITU, 2015); https://www.itu.int/rec/T-REC-G.989.2-201902-I/en
Aijaz, A., Dohler, M., Aghvami, A. H., Friderikos, V. & Frodigh, M. Realizing the tactile internet: haptic communications over next generation 5G cellular networks. IEEE Wirel. Commun. 24, 82–89 (2016).
Takesue, H. et al. Quantum key distribution over a 40-dB channel loss using superconducting single-photon detectors. Nat. Photon. 1, 343–348 (2007).
Torres-Company, V. & Weiner, A. M. Optical frequency comb technology for ultra-broadband radio-frequency photonics. Laser Photon. Rev. 8, 368–393 (2014).
Kuo, B. P.-P., Myslivets, E., Alic, N. & Radic, S. Wavelength multicasting via frequency comb generation in a bandwidth-enhanced fiber optical parametric mixer. J. Lightwave Technol. 29, 3515–3522 (2011).
Zhang, M. et al. Broadband electro-optic frequency comb generation in a lithium niobate microring resonator. Nature 568, 373–377 (2019).
Acknowledgements
We acknowledge financial support from Microsoft, Inphi Inc. and EPSRC grants EP/R041792/1 and EP/R035342/1 and Royal Society Paul Instrument Fund PIF/R1/180001. Eblana Photonics provided the lasers used in this work. We thank P. Watts for helpful discussion at the early stages of the work and E. Vonhof for assistance in the generation of the figures.
Author information
Authors and Affiliations
Contributions
K.A.C., P.C., H.B., I.H., K.J., B.T., H.W. and T.G. conceived the concept of clock phase caching, which was later refined with help from K.S., D.C. and Z.L. K.A.C. and Z.L. conceived and constructed the experimental set-up. K.A.C. implemented clock phase caching in the experiment. K.A.C. designed and implemented all FPGA hardware code. K.A.C. led the experiment and collected all experimental results, supervised by Z.L., with support provided by P.C., H.B., B.T., P.B. and G.Z. P.C. and H.B. collected data-centre traffic and performed the optical switch network utilization analysis. K.A.C., Z.L., P.B., P.C. and H.B. wrote and revised the manuscript. All authors discussed the results and commented on the manuscript.
Corresponding authors
Ethics declarations
Competing interests
A patent application, entitled ‘Phase Caching for Fast Data Recovery’, has been filed by Microsoft Technology Licensing, LLC with the US Patent and Trademark Office on 27 October 2017, on the technology described in this Article. This patent is currently pending (patent no. US20190132112A1).
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Figs. 1–3, Discussion 1–6 and Table 1.
Source data
Source Data Fig. 2
B, Data centre traffic pattern; C, Impact of traffic on throughput.
Source Data Fig. 3
A, 48 hour stability; B, Tolerance to rapid temperature change.
Source Data Fig. 4
A, Impact of rate-of-change of temperature on clock phase caching; B, Minimum required phase update rate as a function of rate-of-change of temperature.
Source Data Fig. 5
A, Impact of sinusoidal jitter on clock phase caching; B, Impact of Gaussian jitter on clock phase caching.
Source Data Fig. 6
Clock phase caching scalability.
Rights and permissions
About this article
Cite this article
Clark, K.A., Cletheroe, D., Gerard, T. et al. Synchronous subnanosecond clock and data recovery for optically switched data centres using clock phase caching. Nat Electron 3, 426–433 (2020). https://doi.org/10.1038/s41928-020-0423-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41928-020-0423-y
This article is cited by
-
Communications with guaranteed bandwidth and low latency using frequency-referenced multiplexing
Nature Electronics (2023)
-
Co-packaged optics (CPO): status, challenges, and solutions
Frontiers of Optoelectronics (2023)