I love Django. I also love PostgreSQL.
I've been fortunate enough to hear Craig Kersteins and Christophe Pettus talk several times about how wonderful Postgres is. I've come away from Django conferences filled with enthusiasm for the promised land of database specificity, of semi-flexible schemas using hstore, of array fields, of json in my database, of full text search and functional indexes and materialized views and… all the other wonderful features offered by Postgres.
Then I go looking for third party packages to use these features. The state of
hstore is pretty good - I can install the
django-hstore package, add a model
and a custom manager to my model and everything works, even the admin. The API
for filtering is pretty good, I get some interesting atomic operations to
update keys via the custom manager, and there's even an interesting subclass
which extends the API to reference other model instances as the values. It's
not all perfect - the API could be more natural in some places (e.g.
.filter(data__mykey=value) rather than
'value'}) - but we are pretty much there.
What about arrays? Well according to django packages there are several
none of which have Python3 support, none of which are currently actively
developed, and most of which don't even have documentation. Alternatively,
there's djorm-ext-pgarray which
is better, but it still requires me to understand how to use array fields in
SQL - I need to know what
@> means, using
SqlExpression objects in
.where() clauses to build my queries. I need to know the postgres name for
the underlying database field I want to use. That's not the ORM I love.
Now, I do not wish to appear too harsh on the developers of these pacakges and
others. Django has in no way helped them make it easy to provide nice APIs. The
custom field syntax is quite limited, only really allowing you to change the
mapping between the database structure and the python structure, and changing
db_type to modify creation. This gets a lot better in Django 1.7 as we
now have custom lookups and
although they currently only work in
.filter() clauses. In particular, I'd
like to thank Andrey Antukh for his work on the
pg-ext-* collection of packages, which have working implementations of most
of the features I wish to build directly into Django.
That said, we can do better. Several people I've worked with, who I consider to be very good Django developers, have absolutely no idea how to go about constructing complex SQL queries by hand. Ask what the difference is between a left and a right outer join and you'll just get a blank expression. Whilst it's easy to be snarky about this and claim it's a failing in their education, I prefer to see it as a huge compliment for the power of the Django ORM. These people have managed to write numerous successful websites with the database at the core of the functionality, without knowing how to interact with their database directly. I would like the more advanced postgres features to have the same ease of use.
Whilst in Warsaw at the wonderful Django Circus conference last summer, I chatted a bit to members of the core team about this concept, and the general state of postgres support in the community. There was agreement that it was not ideal, and we could do better. The idea formed in my head that we could build first class support for these features into Django itself, with the understanding that using database specific functionality is perfectly reasonable (if not required) for any large site.
So I've decided to do something about it. The core team and the DSF have given
their backing for me to do exactly that - build first class support for
PostgreSQL specific features into Django. This will be as a new contrib module
django.contrib.postgres. This is not a small undertaking I can write in an
odd evening or weekend. I'm going to need your help to allow me to spend enough
time on it to make it good. To that end, and inspired by Andrew Godwin's
I'm launching a Kickstarter hopefully at some point in the next week. The
initial aim is to ensure that most Postgres data types have a good Django
equivalent, and for full text search to be possible. With more funding, I'll
also make date based queries much more powerful, add dozens of postgres
specific functions as custom transforms and lookups, and add support for custom
indexes and views. It will take time to build all of this, but I'm confident at
least some features will land for Django 1.8, with most of the rest in Django
For reference, this is what I currently consider as the list of things each field will need to be considered fully supported:
- The field implementation itself, with appropriate keyword arguments to provide the same level of expression as the raw SQL equivalent
- Support for "standard" filters such as
__ltetc if appropriate
- Support for appropriate custom filtering methods specific to the data type
- Where appropriate, a cleverer custom widget for use in the admin
- Extensive tests for each field covering field creation and modification via migrations, all allowable query constructs and validation
- Documentation for the model field and form field, complete with examples and comments on possible use cases.
Thanks for reading, and I hope you are as excited by this project as I am!