Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Synchronous subnanosecond clock and data recovery for optically switched data centres using clock phase caching

Nature Electronics volume 3, pages 426–433 (2020)Cite this article

Subjects

Abstract

The rapid growth in the amount of data being transferred within data centres, combined with the slowdown in Moore’s Law, creates challenges for the future scalability of electronically switched data-centre networks. Optical switches could offer a future-proof alternative, and photonic integration platforms have been demonstrated with nanosecond-scale optical switching times. End-to-end switching time is, however, currently limited by the clock and data recovery time, which typically takes microseconds, removing the benefits of nanosecond optical switching. Here we show that a clock phase caching technique can provide clock and data recovery times of under 625 ps (16 symbols at 25.6 Gb s−1). Our approach uses the measurement and storage of clock phase values in a synchronized network to simplify clock and data recovery versus conventional asynchronous approaches. We demonstrate the capabilities of our technique using a real-time prototype with commercial transceivers and validate its resilience against temperature variation and clock jitter.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Example clock phase cached data-centre architecture, operational principle and demonstration.
Fig. 2: Increase of data centre network utilization enabled by clock phase caching.
Fig. 3: Demonstration of the stability of clock phase caching.
Fig. 4: Impact of rate of change of temperature on clock phase caching.
Fig. 5: Impact of clock jitter on CDR locking time.
Fig. 6: Estimated worst-case optical switch overhead from the clock phase caching technique.

Similar content being viewed by others

Data availability

Source data are provided with this paper.

References

  1. Singh, A. et al. Jupiter rising: a decade of Clos topologies and centralized control in Google’s datacenter network. ACM SIGCOMM Comput. Commun. Rev. 45, 183–197 (2015).

    Article  Google Scholar 

  2. Markov, I. Limits on fundamental limits to computation. Nature 512, 147–154 (2014).

    Article  Google Scholar 

  3. Dorren, H. et al. Challenges for optically enabled high-radix switches for data center networks. J. Lightwave Technol. 33, 1117–1125 (2015).

    Article  Google Scholar 

  4. Ghiasi, A. Large data centers interconnect bottlenecks. Opt. Express 23, 2085–2090 (2015).

    Article  Google Scholar 

  5. Krishnamoorthy, A. et al. From chip to cloud: optical interconnects in engineered systems. J. Lightwave Technol. 35, 3103–3115 (2017).

    Article  Google Scholar 

  6. Testa, F. & Pavesi, L. Optical Switching in Next Generation Data Centers (Springer, 2017).

  7. Ballani, H. et al. Bridging the last mile for optical switching in data centers. In 2018 Optical Fiber Communication Conference (OFC) W1C.3 (OSA, 2018).

  8. Clark, K. et al. Sub-nanosecond clock and data recovery in an optically-switched data centre network. In 2018 European Conference on Optical Communication (ECOC) 1–3 (IEEE, 2018).

  9. Chen, C. P. et al. Programmable dynamically-controlled silicon photonic switch fabric. J. Lightwave Technol. 34, 2952–2958 (2016).

    Article  Google Scholar 

  10. Cheng, Q., Wonfor, A., Wei, J. L., Penty, R. V. & White, I. H. Low-energy, high-performance lossless 8 × 8 SOA switch. In Proc. 2015 Optical Fiber Communication Conference (OFC) Th4E.6 (OSA, 2015).

  11. Shi, K. et al. System demonstration of nanosecond wavelength switching with burst-mode PAM4 transceiver. In Proc. 2019 European Conference on Optical Communication (ECOC) 1–3 (IEEE, 2019).

  12. Xilinx. DS893 (v1.12) Virtex UltraScale Architecture Data Sheet: DC and AC Switching Characteristics (2019); https://www.xilinx.com/support/documentation/data_sheets/ds893-virtex-ultrascale-data-sheet.pdf

  13. Banu, M. & Dunlop, A. E. Clock recovery circuits with instantaneous locking. Electron. Lett. 28, 2127–2130 (1992).

    Article  Google Scholar 

  14. Terada, J. et al. A 10.3 Gb/s burst-mode CDR using a ΔΣ DAC. IEEE J. Solid-State Circuits 43, 2921–2928 (2008).

    Article  Google Scholar 

  15. Rylyakov, A. et al. A 25 Gb/s burst-mode receiver for low latency photonic switch networks. IEEE J. Solid-State Circuits 50, 3120–3132 (2015).

    Article  Google Scholar 

  16. Yang, Y., Wen, Y. J., Nirmalathas, A., Liu, H. F. & Novak, D. Optical clock recovery at line rates via injection locking of a long cavity Fabry–Pérot laser diode. IEEE Photon. Technol. Lett. 16, 1561–1563 (2004).

    Article  Google Scholar 

  17. Ozkaya, I. et al. A 56 Gb/s burst-mode NRZ optical receiver with 6.8 ns power-on and CDR-lock time for adaptive optical links in 14 nm FinFET CMOS. In Proc. 2018 IEEE International Solid-State Circuits Conference (ISSCC) 266–268 (IEEE, 2018).

  18. Bostica, B., Burzio, M., Gambini, P. & Zucchelli, L. in Photonic Networks (ed. Prati, G.) 362–376 (Springer, 1997).

  19. Timing and Synchronization Aspects in Packet Networks G.8261 (ITU, 2008); https://www.itu.int/rec/T-REC-G.8261-201908-I/en

  20. Lee, K. S., Wang, H., Shrivastav, V. & Weatherspoon, H. Globally synchronized time via datacenter networks. In Proc. 2016 ACM SIGCOMM Conference 454–467 (ACM, 2016).

  21. Slavík, R. et al. Ultralow thermal sensitivity of phase and propagation delay in hollow core optical fibres. Sci. Rep. 5, 15447 (2015).

    Article  Google Scholar 

  22. Data Center Networking Equipment—Issues and Best Practices TC9.9 (ASHRAE, 2016); https://tc0909.ashraetcs.org/documents/ASHRAE_TC0909_Power_White_Paper_22_June_2016_REVISED.pdf

  23. Soleimani, S., Afzali-Kusha, A. & Forouzandeh, B. Temperature dependence of propagation delay characteristic in FinFET circuits. In Proc. Int. Conference on Microelectronics (ICM) 276–279 (IEEE, 2008).

  24. Ma, L.-S., Jungner, P., Ye, J. & Hall, J. L. Delivering the same optical frequency at two places: accurate cancellation of phase noise introduced by an optical fiber or other time-varying path. Opt. Lett. 19, 1777–1779 (1994).

    Article  Google Scholar 

  25. Lord, A., Blank, L. C., Boggis, J. M., Bryant, E. & Stallard, W. A. Theory of control mechanism for an optically time-division-multiplexed system. Electron. Lett. 24, 2011–2012 (1988).

    Google Scholar 

  26. Ellis, A. D., Widdowson, T., Phillips, I. D. & Pender, W. A. High speed OTDM networks employing electro-optic modulators. Trans. Inst. Electron. Inf. Commun. Eng. Sect. E E81-C, 1301–1308 (1998).

    Google Scholar 

  27. Zhang, Q., Liu, V., Zeng, H. & Krishnamurthy, A. High-resolution measurement of data center microbursts. In Proc. 2017 Internet Measurement Conference (IMC) 78–85 (ACM, 2017).

  28. IEEE Standard for Ethernet—Amendment 3: Physical Layer Specifications and Management Parameters for 40 Gb/s and 100 Gb/s Operation over Fiber Optic Cables 802.3bm (IEEE, 2015); https://standards.ieee.org/standard/802_3-2018.html

  29. Lipínski, M., Włostowski, T., Serrano, J. & Alvarez, P. White rabbit: a PTP application for robust sub-nanosecond synchronization. In Proc. 2011 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control and Communication (IEEE, 2011); https://doi.org/10.1109/ISPCS.2011.6070148

  30. Bousonville, M. et al. New phase stable optical fiber. In Proc. Beam Instrumentation Workshop 2012 (BIW2012) 101–103 (JACoW, 2012).

  31. 40-Gigabit-Capable Passive Optical Networks 2 (NG-PON2) G.989.2 (ITU, 2015); https://www.itu.int/rec/T-REC-G.989.2-201902-I/en

  32. Aijaz, A., Dohler, M., Aghvami, A. H., Friderikos, V. & Frodigh, M. Realizing the tactile internet: haptic communications over next generation 5G cellular networks. IEEE Wirel. Commun. 24, 82–89 (2016).

    Article  Google Scholar 

  33. Takesue, H. et al. Quantum key distribution over a 40-dB channel loss using superconducting single-photon detectors. Nat. Photon. 1, 343–348 (2007).

    Article  Google Scholar 

  34. Torres-Company, V. & Weiner, A. M. Optical frequency comb technology for ultra-broadband radio-frequency photonics. Laser Photon. Rev. 8, 368–393 (2014).

    Article  Google Scholar 

  35. Kuo, B. P.-P., Myslivets, E., Alic, N. & Radic, S. Wavelength multicasting via frequency comb generation in a bandwidth-enhanced fiber optical parametric mixer. J. Lightwave Technol. 29, 3515–3522 (2011).

    Article  Google Scholar 

  36. Zhang, M. et al. Broadband electro-optic frequency comb generation in a lithium niobate microring resonator. Nature 568, 373–377 (2019).

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge financial support from Microsoft, Inphi Inc. and EPSRC grants EP/R041792/1 and EP/R035342/1 and Royal Society Paul Instrument Fund PIF/R1/180001. Eblana Photonics provided the lasers used in this work. We thank P. Watts for helpful discussion at the early stages of the work and E. Vonhof for assistance in the generation of the figures.

Author information

Authors and Affiliations

  1. Department of Electronic and Electrical Engineering, University College London, London, UK

    Kari A. Clark, Thomas Gerard, Georgios Zervas, Polina Bayvel & Zhixin Liu

  2. Microsoft Research, Cambridge, UK

    Daniel Cletheroe, Istvan Haller, Krzysztof Jozwik, Kai Shi, Benn Thomsen, Hugh Williams, Hitesh Ballani & Paolo Costa

Authors
  1. Kari A. Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Cletheroe
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Gerard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Istvan Haller
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Krzysztof Jozwik
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kai Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Benn Thomsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hugh Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Georgios Zervas
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Hitesh Ballani
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Polina Bayvel
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Paolo Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Zhixin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K.A.C., P.C., H.B., I.H., K.J., B.T., H.W. and T.G. conceived the concept of clock phase caching, which was later refined with help from K.S., D.C. and Z.L. K.A.C. and Z.L. conceived and constructed the experimental set-up. K.A.C. implemented clock phase caching in the experiment. K.A.C. designed and implemented all FPGA hardware code. K.A.C. led the experiment and collected all experimental results, supervised by Z.L., with support provided by P.C., H.B., B.T., P.B. and G.Z. P.C. and H.B. collected data-centre traffic and performed the optical switch network utilization analysis. K.A.C., Z.L., P.B., P.C. and H.B. wrote and revised the manuscript. All authors discussed the results and commented on the manuscript.

Corresponding authors

Correspondence to Kari A. Clark, Paolo Costa or Zhixin Liu.

Ethics declarations

Competing interests

A patent application, entitled ‘Phase Caching for Fast Data Recovery’, has been filed by Microsoft Technology Licensing, LLC with the US Patent and Trademark Office on 27 October 2017, on the technology described in this Article. This patent is currently pending (patent no. US20190132112A1).

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–3, Discussion 1–6 and Table 1.

Source data

Source Data Fig. 2

B, Data centre traffic pattern; C, Impact of traffic on throughput.

Source Data Fig. 3

A, 48 hour stability; B, Tolerance to rapid temperature change.

Source Data Fig. 4

A, Impact of rate-of-change of temperature on clock phase caching; B, Minimum required phase update rate as a function of rate-of-change of temperature.

Source Data Fig. 5

A, Impact of sinusoidal jitter on clock phase caching; B, Impact of Gaussian jitter on clock phase caching.

Source Data Fig. 6

Clock phase caching scalability.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Clark, K.A., Cletheroe, D., Gerard, T. et al. Synchronous subnanosecond clock and data recovery for optically switched data centres using clock phase caching. Nat Electron 3, 426–433 (2020). https://doi.org/10.1038/s41928-020-0423-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41928-020-0423-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing