EveryBlock

A news feed for your block — Saturday, 3rd January, 2009

Adrian Holovaty is no stranger to Django, and his name is known throughout the community as one of the brains behind Django's birth and subsequent open-source release. His latest project EveryBlock is the evolution of an earlier mashup, chicagocrime.org, which won Adrian a number of awards. So what are the driving forces behind EveryBlock? I recently spoke with Adrian to find out.

Vital Stats

11 major US cities covered down to the street level
Custom mapping using mapnik
Strong use of Django's caching and nginx reverse-proxying mechanisms to manage loads
Development done by a team of 3 developers, plus a designer

The Down-Low

As one of Django's founders and core developers, Adrian Holovaty is known in Django circles as the friendly BDFL who plays the guitar as well as he codes. One of his earliest Django projects outside the Lawrence Journal-World (where he began the Django project with Simon Willison was chicagocrime.org, a mash-up that showed crime details from the City of Chicago Police Department overlaid on Google Maps. It was a unique idea that spawned a number of Google Maps-based mash-ups.

However he had bigger targets in mind. In January of 2008, EveryBlock was opened to the world.

Behind it was not only Adrian's experience with chicagocrime.org and Django, but developer Paul Smith and designer Wilson Miner. What the visitors saw was also quite different: Not only had the EveryBlock team aggregated crime statistics for a number of cities, but other public information was also added to the foray including news, property foreclosures, business listings, and photographs.

EveryBlock aims to show you practically everything that's possible about the goings on near a city block, by combining data aggregated from multiple sources with custom city maps. It's built in Django, with Adrian's only concern being that it cannot correctly pluralise "octopus". Nothing else was considered, for somewhat obvious reasons.

From the projects beginnings in July 2007, it took the team 6 months to launch the public website, although Adrian says "that time includes a significant amount of effort spent on acquiring data for those initial cities". After the launch, the development team was expanded to include Joseph Kocherhans.

When EveryBlock was started, GeoDjango was still in its infancy so the team implemented what Adrian calls "fancy geographic extensions" on top of an otherwise straightforward Django codebase. To deal with the complex calculations required for some pages, high-traffic pages are served directly from memcache by nginx instead of having visitors repeatedly hit the application.

EveryBlock also optimises memory use by making extensive use of the ORM's values() function, avoiding the overhead of creating python model objects. This provides the developer with a plain Python dictionary of values which is suitable in many cases.

All of this data is presented to the user overlaid on custom city maps, built using OpenLayers and Mapnik. This solid open-source core is extended with the use of software such as the nginx web server, Monit (for keeping a watchful eye on servers), and the PostgreSQL database.

In terms of Django, EveryBlock runs against a recent subversion checkout without any patches. Unlike many projects which end up with a small repository of their own patches to the core product, Adrian is in the unique situation of being able to fix problems in Django as he discovers them. He's also got a better-than-average understanding of many of Django's internals, allowing for code that steps outside the lines of what's documented.

As EveryBlock grows and continues to add cities to their listing, very little coding is required. "Adding a new city requires adding the data, which is the really time-consuming part. The work pays off. Nobody else is crazy enough to do it!".

And the Future? The main aim right now "More cities, more data in existing cities, and more features!". And of course, it's going to be powered by Django.

A big thank-you to Adrian Holovaty from EveryBlock for taking time out of his busy schedule to answer my questions for this week's Django Site of the Week.

Thanks

Special thanks go to Adrian Holovaty for this interview.

EveryBlock can be visited at http://www.everyblock.com/. It is also listed at DjangoSites.org.

Comments

Posted by Agitationist

7:43 a.m. on Wednesday, 28th January, 2009

EveryBlock is an absolutely superb tool - a must for anyone who wants to keep up on their neighborhood. More here: http://agitationist.com/why-everyblock-rules-the-neighborhood

Posted by KIRSTENhg20

8:08 p.m. on Tuesday, 9th February, 2010

Different people that would like to have high grades find a professional <a href="http://quality-papers.com">essay writing</a> service to purchase their persuasive term paper at. That things we do as well.