Lost your password?
Click here

ForecastWatch and the U.S. Government Shutdown

As most of you know, the United States entered a "partial shutdown" on October 1. Leaving politics aside I wanted to discuss with you how this affects ForecastWatch. ForecastWatch receives observational data generated by entities partially or wholly funded by the United States Federal government that have been affected by this shutdown. First, ForecastWatch uses the Quality-Controlled Local Climatic Data (QCLCD) product from the National Climatic Data Center for United States Observations. This data set is well-maintained, clean, and provides high resolution (5-minute) 24-hour high and low. There are a number of other qualities that make it a great choice for ForecastWatch, and we've used the LCD product in various forms since 2004 for U.S. observations. It used to be that we paid $500/year for this dataset, and it was accessed via FTP from specific IP addresses associated with the order. A few years ago, FTP access was discontinued for authenticated HTTP. Finally, in 2012, the dataset became free. Internationally (save Canada), we use the Integrated Surface Database (ISD), also provided by the NCDC. It is an amazing complete and clean dataset. We used to pay $2000/year for this dataset, and was accessed via username/password FTP. Last year, it also became free, and available via anonymous FTP. (My only editorializing: I was concerned when the both datasets became free, I would have gladly paid, because when you don't have customers you don't have obligations.) Because all NCDC HTTP sites now redirect to a "government shutdown" placard (http://governmentshutdown.noaa.gov/) we no longer have access to the QCLCD product. However, the FTP site is still up and running. (Ok, more editorializing, in an example of the arbitrariness of that "shutdown", the NCDC FTP site is still available but the HTTP sites redirected because "they can't be maintained".) Because the FTP site is still available, we can access the ISD data, however, it was last updated September 30, the day before the shutdown, so it doesn't contain the full month of September, as observations often lag a day or two behind. ForecastWatch currently receives Canadian observations from Environment Canada, which is still working peachy. Yeay Canada! So how does this affect ForecastWatch? 1. Canadian aggregations for September will be unaffected and delivered on schedule. 2. International aggregations for September will be missing between a day and a week of observations at the end of September. 3. U.S. aggregations for September will be delayed. Please write your congress-person. What mitigation is happening? 1. For U.S., we are looking at pulling U.S. observational data from ISD so at least we have a nearly-complete set. The major difference is we don't get a 5-minute resolution on the 24-hour high and low. There may be other differences, and it is far from clear if it can be done seamlessly and in a reasonable time. 2. I have tried contacting NCDC for alternative access to the QCLCD data, but as expected haven't received a response. A partial QCLCD file for September would at least provide something. If anyone has any informal channels/contacts within NCDC that could identify a non-HTTP way to access QCLCD, that would be great. 3. Once the shutdown has been resolved and the observational data for September (both QCLCD and ISD) complete, we'll rerun the September aggregations with complete data.