• Latest
• All Sites
• Why?
• Suggest

ForecastWatch

Improve Your Weather Forecast Accuracy — Tuesday, 27th January, 2009

Eric Floehr is the man behind ForecastWatch and ForecastAdvisor, two Django-powered weather websites that aggregate and analyse weather forecasts to compare their accuracy on an ongoing basis. This week, I spoke to Eric about the history behind his sites, how he handles massive data sets, and his conversion from Ruby on Rails to Django.

Vital Stats

• A dual-convertee, moving Quixote and Ruby on Rails projects to Django
• 120 million weather forecasts
• 1.2 million observations
• Used by commercial entities to find the accuracy of their weather forecasts.

The Down-Low

Getting into computers at a young age, Eric has had a lifelong interest in science, the environment and the weather. This interest in the weather led to a pair of recurring questions: Do different providers give different forecasts? And how accurate are they? About five years ago, he decided to indulge his curiosities and built ForecastWatch and ForecastAdvisor.

Originally built with Quixote, ForecastWatch was covered as a Python Success Story back in 2004. Today, it's used by a wide range of customers interested in weather forecast accuracy including energy companies, transportation departments, media companies, and weather companies competing against each other. From day one, it's been entirely written in Python - everything from the data scraping and aggregation to web presentation is written in Python.

Eric began to transition ForecastWatch to Django about three years ago, before Django's magic-removal. When the site was initially built, Quixote was chosen because it was powering real-world websites however it's limitations as a "first generation" web framework soon made managing the website more of a chore than it should have been. "Django just made things a lot easier", says Eric fondly about his move to Django.

On the other hand, ForecastAdvisor was originally a Ruby on Rails project designed to provide consumer-level access to the ForecastWatch data for those wanting to see just a small subsecton of the data without paying ForecastWatch's commercial rates. At the time the website was built, Django was not yet known outside of a small newspaper in Lawrence, Kansas and Rails was building plenty of buzz online. Although Eric has been a Python user for almost 8 years, as with any technology enthusiast he's "not so myopic to not try new technologies when warranted." This led him to try out Ruby on Rails, which powered the website until he discovered the pythonic nature of Django.

One of the big drivers for moving from Ruby on Rails to Django was the volume of 'magic' in Rails which was being removed from Django at the time. Being a programmer who really likes to know what's going on under the hood, the magic present in both Rails and Django at the time frustrated Eric. The differences between developing with the two frameworks didn't go much deeper than code syntax, though. Eric found that the conversion was surprisingly easy, noting many similarities between the two frameworks. A mentor introduced him to the development methods now known as agile development over a decade ago, and both Django and Ruby on Rails allowed Eric to continue using those methods.

Now using a stable checkout from Subversion, Eric has received a number of improvements to earlier versions of Django such as the 'magic removal' from 2006 and a cleanup of the URL handling mechanisms to allow URL's to be defined in a single location. He's also a big fan of the ORM and Model structure within Django, and makes extensive use of the Model API outside of web applications for data input and manipulation.

Although data aggregation has recently made it into Django's core, Eric processes his aggregation offline on a scheduled basis due to the volumes of data involved. The query-time SQL aggregation functions that are used by Django's ORM would not allow the site to be as responsive as it currently is, so he makes the most of the fact that disk space is a cheaper alternative to raw processing power. With over 120 million forecasts and 1.2 million observations on file, the monthly aggregation tasks require about a day of processing raw data into aggregation tables, plus another half-day to build the relevant data maps showing high & low temperatures and other statistics.

As with many Django projects, a number of open-source applications keep ForecastWatch up and running. Servers are a combination of CentOS and Fedora, running MySQL databases and lighttpd to serve web data. virtualenv has helped with deployment, and although it isn't open source WingIDE is Eric's IDE of choice.

So what does the future hold other than ongoing development for both ForecastWatch and ForecastAdvisor? With increased interest in global warming and climate change, Eric is currently working on a new project to help educate, entertain and bring people together while documenting our changing climate over time. Will he be using Django? "Absolutely. As you go deeper, Django goes with you, allowing you to do what you want without getting in the way."

Thanks

Special thanks go to Eric Floehr for this interview.

ForecastWatch can be visited at http://forecastwatch.com/.

Comments

Share Your 2c

Name

Email address

This is not published, and is only used for moderation purposes.

URL

Comment

Markdown is hot, HTML is not.

If you enter anything in this field your comment will be treated as spam

© Copyright 2008 Ross Poulton. All Rights Reserved. Django is a registered trademark of the Django Software Foundation.