Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

DATA COMPRESSION

Advancing data compression via noise detection

Nature Computational Science volume 1, pages 711–712 (2021)Cite this article

Subjects

Compressing scientific data is essential to save on storage space, but doing so effectively while ensuring that the conclusions from the data are not affected remains a challenging task. A recent paper proposes a new method to identify numerical noise from floating-point atmospheric data, which can lead to a more effective compression.

Checking the weather forecast with an application (or app) on one’s phone is a critical part of many morning routines. For example, when making the decision of whether to wear pants or shorts on a sunny day in Colorado, if the app reports that the day’s high will be 50 degrees Fahrenheit (°F), most people will potentially opt for pants. But what if the app reports 50.456783957642 °F for the high? Probably still pants. For most of us, twelve extra decimal places of information are unlikely to influence this decision. Weather (and climate) forecasts come from complex model simulation codes, and while simulation codes are an important tool in scientific discovery, one must put the information that they output into context. If the weather app reported that today’s high would be 50.456783957642 °F, a few issues would come to mind. First, do we need that much precision for the data to be useful? The precision probably depends on how we will use the data. Second, are all these digits meaningful? Do they contain real information? Because computers store and operate on data in predetermined sizes, generally 8 bytes (double-precision) or 4 bytes (single precision), for many simulation codes a lot of that information is either not necessary for its intended use (meaning, the weather app) or the trailing digits may just be numerical noise (called false precision by Zender1). The bottom line is that models often use a higher precision than what is physically important (for instance, storing temperature as 64 bits), and precision does not necessarily equal practically useful information. Writing in Nature Computational Science, Milan Klöwer and colleagues2 propose a method that automatically determines noise in large atmospheric data, leading the way to advance compression of large scientific datasets from numerical models.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The notion of noise and precision.

References

  1. Zender, C. S. Geosci. Mod. Dev. 9, 3199–3211 (2016).

    Article  Google Scholar 

  2. Klöwer, M., Razinger, M., Dominguez, J. J., Düben, P. D. & Palmer, T. N. Nat. Comput. Sci. https://doi.org/10.1038/s43588-021-00156-2 (2021).

  3. Lindstrom, P. Error distributions of lossy floating-point compressors. In JSM Proc. 2574–2589 (2017).

  4. Lindstrom, P. & Isenburg, M. IEEE Trans. Vis. Comput. Graph. 12, 1245–1250 (2006).

    Article  Google Scholar 

  5. Baker, A. H. et al. Geosci. Mod. Dev. 9, 4381–4403 (2016).

    Article  Google Scholar 

  6. Poppick, A. J. et al. Comput. Geosci. 145, 104599 (2020).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Mathematics and Statistics, Colorado School of Mines, Golden, CO, USA

    Dorit M. Hammerling

  2. Computational and Information Systems Laboratory, National Center for Atmospheric Research, Boulder, CO, USA

    Allison H. Baker

Authors
  1. Dorit M. Hammerling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Allison H. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dorit M. Hammerling.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hammerling, D.M., Baker, A.H. Advancing data compression via noise detection. Nat Comput Sci 1, 711–712 (2021). https://doi.org/10.1038/s43588-021-00167-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s43588-021-00167-z

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing