Tom Coote » Django

Django Site Login Script

Tom Coote — Sun, 08 Jan 2012 14:18:38 +0000

Just having a play around with a Python script that will login to a standard Django login form. I thought I’d share it as it’s quite interesting and I’m sure someone might find a use for it.

It uses the standard Python modules urllib2, urllib and cookielib. It also requires BeautifulSoup for getting the generated CSRF token from the Django login form to use with the authentication credentials.

Here is the script.

import urllib2
import urllib
import cookielib
from BeautifulSoup import BeautifulSoup

LOGIN_URL = 'http://a-django-site.com/login/'

# for handling cookies to and from all requests we make in this script
# because the site uses cookies for authentication data.
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))

# need the CSRF token from the login form that allows us to login
login_form = BeautifulSoup(opener.open(LOGIN_URL).read())
csrf = login_form.find('input', attrs={'name': 'csrfmiddlewaretoken'}).attrMap

# sign in by sending the correct credentials to the login form
credentials = {'username': 'admin',
               'password': '123456',
               'remember': '',
               'csrfmiddlewaretoken': csrf['value']
               }

req = urllib2.Request(LOGIN_URL, urllib.urlencode(credentials))
raw_html = opener.open(req).read()

# test if the login was successful
login_success = 'sessionid' in [c.name for c in cj]
print login_success and 'Login successful' or 'Login failed'

Pretty straight forward. Once logged in BeautifulSoup could be used to easily navigate and scrape data from the authenticated web site.

Improve Django SQL Queries Using Debug Toolbar

Tom Coote — Sun, 09 Oct 2011 21:51:52 +0000

Django’s database querying is there to make life easier but it’s also easy to forget about the SQL that is still generated under the hood. When using Django’s query objects it’s extremely difficult to know how often the database is being hit and with what SQL queries. We often write queries in our views, check that they return what we want, then forget them. Lots of the time that is okay but often we’ll run into performance problems later down the road due to too many database queries. Even if you only have one query in your view, the template might generate more queries without you knowing about them.

Luckily we have the django debug toolbar to show us how many queries are being sent to our database for a single page request. So I’m going to go through an example of using the debug toolbar to make a performance improvement for a django view. I’m not going into how to install the debug tool bar because it’s all on their download page here.

I created a simple test site to demonstrate how to find and improve database hits. It has one page which just lists a load of comments from the database. The template HTML is as follows.




	Debug Demo

	



	Comments

	{% for comment in comments %}
	
	Posted by: {{ comment.posted_by.first_name }}
	Date: {{ comment.post_date|date:"D jS M" }}

	{{ comment.text|linebreaks }}
	
	{% endfor %}

So as you can see, the template is expecting a list of comments and is just going to create markup to display each of them. The Django model and view are as follows.

from django.db import models
from django.contrib.auth.models import User

class Comment(models.Model):
    post_date = models.DateTimeField(auto_now_add=True)
    posted_by = models.ForeignKey(User)
    text = models.TextField()

from django.shortcuts import render
from DebugDemo.models import Comment

def comments(request):

    return render(request, 'comments.html', {
                     'comments': Comment.objects.all()
                    })

The result is as expected, the page loads and we see all comments in the database. What you may think by looking at our view code is that one database hit is made, just one that gets all comments. In reality we have one hit to get the comments plus one hit per comment that is retrieved. So to put this in Python form we have len(comments)+1 database hits. The problem is that we won’t notice a performance problem for this until we have loads of comments to show and loads of visitors wanting to see them. We can see the SQL that was sent to the database by using the debug toolbar and opening the SQL tab. In my example I had two comments in the database so I had three queries as shown below.

What’s happening here is that the view will run one database query to retrieve all the comment data then our template needs the user’s name to show against each comment, so when it get’s to that it will do another query to retrieve each user for each comment. Understanding why these queries are generated is half the battle but it’s made a lot easier with the debug toolbar as we can see what the SQL is and therefor know what data each is retrieving. When our list of SQL queries has one for comments and loads more for users it’s pretty obvious what’s going on.

Solving this is a matter of re-writing our view query so that it retrieves the data that our template needs all at once. To do that we can use the query objects values method. It’s always about not retrieving the things you don’t need. So our new view looks like this.

from django.shortcuts import render
from DebugDemo.models import Comment

def comments(request):

    comments = Comment.objects.all().values('post_date', 'posted_by__first_name', 'text')

    return render(request, 'comments.html', {
                     'comments': comments
                    })

You can see that we’re using the values method to tell Django to only retrieve the three columns that our template needs. The problem now is that our template needs a bit of an update. It currently calls comment.posted_by.first_name to print the users name but this is no longer correct because our new query returns a dictionary of key value pairs not a list of model objects. So here is the new template code, it’s just using comments.posted_by__first_name instead.




	Debug Demo

	



	Comments

	{% for comment in comments %}
	
	Posted by: {{ comment.posted_by__first_name }}
	Date: {{ comment.post_date|date:"D jS M" }}

	{{ comment.text|linebreaks }}
	
	{% endfor %}

The result is just one query which we can see in the debug toolbar.

I hope I’ve shown how important it is to keep on eye on the queries that are going on because even in this really simple example we can see how performance problems can easily creep in. My advice is to always run through the pages you build with the debug toolbar to have a quick look at the queries to see if there is anything unexpected going on.

Django Class Based Generic API

Tom Coote — Sun, 12 Jun 2011 14:06:47 +0000

It has been a pretty rainy weekend so I’ve decided to do some Python programming and have come up with a set of tools to help build API’s in django.

I know there are a few libraries already available for building API’s in django (I’ve used and contributed to some of them) but since django 1.3 was released with the class based generic views then a lot of the functionality needed to create API’s is already in the framework. So I decided to just build a few extra things to make API’s easy in django by using the framework for most of the heavy lifting.

I’ve made the code available on GitHub. Here are the features in a nut shell;

A new generic class view called APIView which provides a mini framework for supporting different data formats for an API and providing some common API responses.
A throttling decorator to limit API calls per user.
Support for base64 http authentication.

I’ve put basic support in for JSON requests and responses. The code acts as a good basis for a complete API in django and allows an easy way to extend it for new data formats. It makes good use of POST, GET, PUT and DELETE request methods.

View the repository on GitHub. I would be interested to know how anyone gets on using it and if anyone adds in support for different data formats.

Django South in a team using SVN

Tom Coote — Fri, 26 Nov 2010 19:57:25 +0000

South, the schema and data migration tool for django is really useful. I’ve only started using it myself recently but the benefit is immediate. South creates python files which hold all the information for migrations each time you change the database schema. Each person in a team of developers needs to run these migrations on their development database to keep things up to date and working.

The question is how does one person in a team know that another has changed the schema? If time has passed then simply asking could get a somewhat hazy response.

We use SVN to keep track of code changes amongst the team of developers. SVN is a good place to look to see if migrations were committed. The only thing better than looking in SVN logs for migrations is for SVN to tell us when migrations are committed. So that is what I’ve had a go at doing.

Using the SVN pre-commit hook I’ve been able to send emails to those involved in a project each time someone commits code that contains new database migration changes. The first part of this was to create a pre-commit.bat file in the hooks directory of my projects repository.

C:\python26\python.exe D:\repositories\post_commit_emailer.py %* tom@email.com bob@email.com eddy@email.com

This batch file is run by SVN just before a commit happens. It is passed the repository and transaction name when called. As you can see, I’ve also appended some extra parameters which are the emails of everyone involved with this project.

So the next bit is the python script that this batch file calls.

import smtplib
import sys
import pysvn
import re
from datetime import datetime

SERVER = 'localhost'
FROM = 'svn@email.co.uk'
SUBJECT = 'SVN South Migrations Committed'
TEXT = \
'''
The repository location '%s' was committed and included database schema migrations.

Next time you update your working copy for this repository be sure to run the new south migrations on your local database.

The changed files that are thought to be schema migrations are:
   - %s
'''

try:
    # what was committed?
    repo = sys.argv[1]
    txn = sys.argv[2]
    transaction = pysvn.Transaction(repo, txn)
    changes = transaction.changed()

    # look for database migrations that may have been part of this commit
    migrations = []
    for f in changes.keys():
        if re.search(r'migrations/.*\.py$', f):
            migrations.append(f)

    if migrations:
        # prepare the message. Emails to send to are passed as args
        emails = sys.argv[3:]
        message = TEXT % (repo, '\n   - '.join(migrations))
        message = "From: %s\nTo: %s\nSubject: %s\n\n%s" % \
                    (FROM, ', '.join(emails), SUBJECT, message)

        # send the email
        server = smtplib.SMTP(SERVER)
        server.sendmail(FROM, emails, message)
        server.quit()

except Exception as e:
    log = open('C:\\svn_emailer_log.txt', 'a')
    log.write('\n%s : %s' % (datetime.now().isoformat(), str(e)))
    log.close()

This script uses pysvn to grab all the changes that are to be committed in this transaction. Once it has that information it looks at the path of each file in the list of changes to find any in a migrations folder. South puts migration files in a folder named migrations. If it finds any then it composes an email to everyone in the parameter list. The email lists the migrations that were committed.

So now everyone involved in a project will get an email if someone commits database schema changes. I’m going to start using this for our django projects and see how it goes.

Global Django Permissions

Tom Coote — Sun, 31 Oct 2010 18:20:07 +0000

Django has a great solution for web site permissions, it allows the developer to decide how a user can interact with an application based on permissions against models (i.e. database tables). By default a model comes with three permissions, add, change and delete. If these aren’t enough then you can create as many custom model based permissions as needed very easily. Infact, this easily:

permissions = (("can_deliver_pizzas", "Can deliver pizzas"),)

This is great but what if your app doesn’t have a model. Without a model your app doesn’t get the three default permissions and you can’t create any custom permissions because you don’t have the model to base them upon. Well the solution is a simple one but maybe not an obvious one. Django holds two tables for it’s permissions, one called ContentType and one called Permission. You need to create a new ContentType entry for your app and then add as many entries to the Permission table as you need. The ContentType table has a column for the model name but it’s just fine to leave that blank.

The next question though is where to put the code to do this job? Ideally it needs to go in a place that is parsed when syncdb is run but not in a place that continually get’s called as it would have a negative impact on performance. Since the whole point of creating these permissions is because we aren’t using models, no other part of our app will ever try and import the models file. So the models file is the place to put our code. This file is parsed on syncdb but after that will never be called upon again.

So great, we have the method and we have the place to write the code. All that’s left is the code itself which is below. We’re creating one ContentType with the name of our app and then as many Permission’s as needed, in this case just one!

from django.contrib.auth.models import Permission
from django.contrib.contenttypes.models import ContentType
from django.db.models.signals import post_save

def create_app_permissions(sender, **kwargs):
	# create a content type for the app if it doesn't already exist
	ct, created = ContentType.objects.get_or_create(model = '', app_label = 'myappname',
	defaults = {'name': 'myappname'})

	# create a permission if it doesn't already exist
	Permission.objects.get_or_create(codename = 'can_deliver_pizza',
	content_type__pk = ct.id,
	defaults = {'name': 'Can Deliver Pizza', 'content_type': ct})

post_save.connect(create_app_permissions, Permission)

One last thing. This code uses signals to run once the Permission model has saved. This means the code won’t crash when using syncdb with complaints about the Permission or ContentType tables not existing because it won’t get run until django has created it’s permissions.

Django using Eclipse IDE and Pydev

Tom Coote — Sat, 24 Jul 2010 12:38:44 +0000

I’ve been using Wingwares python IDE for Django development for almost 2 years. The Wingware IDE has been very good but I have had a few superficial annoyances with it so decided to look around and see what other IDE’s I could use. This led me to Eclipse. The first time I had a look around for Django IDE’s I remember looking at Eclipse but I think I recall difficulties when trying to run Django projects. Eclipse doesn’t natively support Django, you have to install Pydev to get that. After revisiting Eclipse and Pydev I can safely say I’ve switched from Wingware to Eclipse for Django development. The Eclipse IDE in itself is very good and full of useful development features. The Pydev plugin for it which allows python and Django development is also fantastic as it has a GUI for creating and managing Django specific python projects. Another great feature of Eclipse is it’s free price tag where as a single developer license for Wingware is $179 (roughly £115).

To get my Eclipse set up for Django I installed the following:

Then I read through and followed a few guides to understand Pydev and Django support:

In less than an hour I was up and running and fully understood how to start and develop Django projects in Eclipse. I’d even moved 3 existing Django projects into Eclipse in that time.

Sorry Wingware, you’ve been good to me but I’m trying Eclipse for a while and I don’t think I’m coming back.