As part of our ongoing transition to modern, cloud-native infrastructure in AWS, we are continuously evaluating how to best deliver high-volume geophysical data to researchers, educators, and operational partners around the globe. Operating in the cloud provides exceptional scalability and performance, and we want to guide users to data and product access methods that are appropriate for the work they do.
Most users download data and products to their local computers using workflows that have been the mainstay of seismic and geodetic research for years. We expect that this practice will continue, as will our services that support download. However, to take full advantage of the speed and scalability of the cloud, which can make a huge difference when processing terabytes of data, direct access is the best option.
While we believe that direct data access is the best mechanism for all users, we are guiding large data users away from the traditional access mechanisms by implementing limits on the volume of data and products that can be downloaded per hour and per day over the internet. The limits for registered users will be much higher than the limits for anonymous users.
We want to share what to expect, how high-volume users can prepare, and the powerful alternative access pathways available today.
What Is Changing?
On October 26, the amount of data that can be downloaded to the internet from the NSF NGF will be set to 200 GB/hour and 1,000 GB/day for registered users, and to 10 GB/hour and 50 GB/day for anonymous users. These limits accommodate most of the requests that we receive. We encourage all users to become registered users, as anonymous data access will not be permitted after December 15, 2026. No limits will be placed on direct S3 data access via the Open Data Program or from a cluster within the us-east-2 AWS region.
If the volume of information requested exceeds hourly or daily download thresholds, further access will be temporarily denied:
- HTTP 429 Responses for Limit Exceedance: Heavy users who exceed the hourly or daily download volume ceiling will receive an HTTP 429 Too Many Requests status code. In standard HTTP conventions, a 429 indicates that a client has transmitted too much data or sent too many requests within a given time window.
- Duration of Temporary Blocks: When a limit is tripped, access will be temporarily restricted:
- Hourly threshold exceeded: Restricted until the top of the next hour.
- Daily threshold exceeded: Restricted until midnight (UTC), resetting for the new calendar day.
High-Volume Alternatives: Direct S3 Access and Open Data Program
For research workflows requiring hundreds to thousands of gigabytes of data—or complete repository mirrors—traditional web service endpoints like fdsnws/dataselect are not an efficient mechanism. We offer two robust alternatives designed specifically for bulk access:
- AWS Open Data Sponsorship Program: Access several datasets that are hosted via the AWS Open Data Program (ODP). Note that the ODP currently cannot host the entire archive, but it is nonetheless a strong offering.
- In-Region Direct S3 Access: For cloud-native computing, workflows running within AWS us-east-2 can directly stream miniSEED and other data objects straight from S3 at maximum throughput and minimal latency.
Both pathways allow you to conduct large-scale computational experiments efficiently.
Looking Ahead & Sharing Feedback
We value your partnership and welcome your feedback, questions, or use-case inquiries as we roll out these changes. Reach us through our help desk at help@earthscope.org.