NOAA Big Data Project

The National Oceanic and Atmospheric Administration (NOAA) Big Data Project (BDP) was created as an approach to publishing the agency’s vast data resources with cost-efficient high performance computing, analytic and storage services.  Working with private sector cloud infrastructure providers Amazon Web Services, Google, IBM, Microsoft, and the Open Commons Consortium, “This collaboration is designed to facilitate full and open data access at no net cost to the taxpayer, and to foster innovation by bringing together the tools necessary to make NOAA's data more readily accessible.”

NOAA collects data from the bottom of the ocean to the surface of the sun. The data comes from 10 satellites, more than 150 weather radars, 3 buoy networks, over 200 tide gauges, human observers, ships, aircraft, and more. NOAA gathers more than 20 terabytes of data every day – adding up to more than 6 petabytes by 2014 – and user demand for access to that data is growing rapidly. But despite the growing demand, not all of NOAA’s data is easily accessible to the public, especially in the case of very large datasets.

To change this, NOAA is partnering with private sector organizations that have the infrastructure and technical capacity to deal with the volume and complexity of NOAA’s data. These organizations host selected NOAA datasets in the cloud, providing users remote access along with the ability to perform computations directly on the data.  These partnerships are officially established through Cooperative Research and Development Agreements (CRADA’s), a vehicle for collaboration that other research-driven agencies can also use.