Tom Coote » Tom Coote

Django Site Login Script

Tom Coote — Sun, 08 Jan 2012 14:18:38 +0000

Just having a play around with a Python script that will login to a standard Django login form. I thought I’d share it as it’s quite interesting and I’m sure someone might find a use for it.

It uses the standard Python modules urllib2, urllib and cookielib. It also requires BeautifulSoup for getting the generated CSRF token from the Django login form to use with the authentication credentials.

Here is the script.

import urllib2
import urllib
import cookielib
from BeautifulSoup import BeautifulSoup

LOGIN_URL = 'http://a-django-site.com/login/'

# for handling cookies to and from all requests we make in this script
# because the site uses cookies for authentication data.
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))

# need the CSRF token from the login form that allows us to login
login_form = BeautifulSoup(opener.open(LOGIN_URL).read())
csrf = login_form.find('input', attrs={'name': 'csrfmiddlewaretoken'}).attrMap

# sign in by sending the correct credentials to the login form
credentials = {'username': 'admin',
               'password': '123456',
               'remember': '',
               'csrfmiddlewaretoken': csrf['value']
               }

req = urllib2.Request(LOGIN_URL, urllib.urlencode(credentials))
raw_html = opener.open(req).read()

# test if the login was successful
login_success = 'sessionid' in [c.name for c in cj]
print login_success and 'Login successful' or 'Login failed'

Pretty straight forward. Once logged in BeautifulSoup could be used to easily navigate and scrape data from the authenticated web site.

Improve Django SQL Queries Using Debug Toolbar

Tom Coote — Sun, 09 Oct 2011 21:51:52 +0000

Django’s database querying is there to make life easier but it’s also easy to forget about the SQL that is still generated under the hood. When using Django’s query objects it’s extremely difficult to know how often the database is being hit and with what SQL queries. We often write queries in our views, check that they return what we want, then forget them. Lots of the time that is okay but often we’ll run into performance problems later down the road due to too many database queries. Even if you only have one query in your view, the template might generate more queries without you knowing about them.

Luckily we have the django debug toolbar to show us how many queries are being sent to our database for a single page request. So I’m going to go through an example of using the debug toolbar to make a performance improvement for a django view. I’m not going into how to install the debug tool bar because it’s all on their download page here.

I created a simple test site to demonstrate how to find and improve database hits. It has one page which just lists a load of comments from the database. The template HTML is as follows.




	Debug Demo

	



	Comments

	{% for comment in comments %}
	
	Posted by: {{ comment.posted_by.first_name }}
	Date: {{ comment.post_date|date:"D jS M" }}

	{{ comment.text|linebreaks }}
	
	{% endfor %}

So as you can see, the template is expecting a list of comments and is just going to create markup to display each of them. The Django model and view are as follows.

from django.db import models
from django.contrib.auth.models import User

class Comment(models.Model):
    post_date = models.DateTimeField(auto_now_add=True)
    posted_by = models.ForeignKey(User)
    text = models.TextField()

from django.shortcuts import render
from DebugDemo.models import Comment

def comments(request):

    return render(request, 'comments.html', {
                     'comments': Comment.objects.all()
                    })

The result is as expected, the page loads and we see all comments in the database. What you may think by looking at our view code is that one database hit is made, just one that gets all comments. In reality we have one hit to get the comments plus one hit per comment that is retrieved. So to put this in Python form we have len(comments)+1 database hits. The problem is that we won’t notice a performance problem for this until we have loads of comments to show and loads of visitors wanting to see them. We can see the SQL that was sent to the database by using the debug toolbar and opening the SQL tab. In my example I had two comments in the database so I had three queries as shown below.

What’s happening here is that the view will run one database query to retrieve all the comment data then our template needs the user’s name to show against each comment, so when it get’s to that it will do another query to retrieve each user for each comment. Understanding why these queries are generated is half the battle but it’s made a lot easier with the debug toolbar as we can see what the SQL is and therefor know what data each is retrieving. When our list of SQL queries has one for comments and loads more for users it’s pretty obvious what’s going on.

Solving this is a matter of re-writing our view query so that it retrieves the data that our template needs all at once. To do that we can use the query objects values method. It’s always about not retrieving the things you don’t need. So our new view looks like this.

from django.shortcuts import render
from DebugDemo.models import Comment

def comments(request):

    comments = Comment.objects.all().values('post_date', 'posted_by__first_name', 'text')

    return render(request, 'comments.html', {
                     'comments': comments
                    })

You can see that we’re using the values method to tell Django to only retrieve the three columns that our template needs. The problem now is that our template needs a bit of an update. It currently calls comment.posted_by.first_name to print the users name but this is no longer correct because our new query returns a dictionary of key value pairs not a list of model objects. So here is the new template code, it’s just using comments.posted_by__first_name instead.




	Debug Demo

	



	Comments

	{% for comment in comments %}
	
	Posted by: {{ comment.posted_by__first_name }}
	Date: {{ comment.post_date|date:"D jS M" }}

	{{ comment.text|linebreaks }}
	
	{% endfor %}

The result is just one query which we can see in the debug toolbar.

I hope I’ve shown how important it is to keep on eye on the queries that are going on because even in this really simple example we can see how performance problems can easily creep in. My advice is to always run through the pages you build with the debug toolbar to have a quick look at the queries to see if there is anything unexpected going on.

Writing functions inside loops in JavaScript

Tom Coote — Sat, 08 Oct 2011 16:57:58 +0000

I’ve created a short video to help explain what the issues are when writing JavaScript functions inside loops and how to use function scope to solve those issues. Enjoy.

Full screen it in order to read the code.

Code To Give Users Time To Read Feedback

Tom Coote — Wed, 13 Jul 2011 13:42:58 +0000

I’ve blogged before about giving users of a web application feedback on the page whilst they wait for something to load and also about slowing down the responses from AJAX requests in order to give those users time to fully read and understand that feedback. I’ve recently built some more code to help achieve this. This is a jQuery extension.

If you’re going to use AJAX to load some content into the page as the user requests it then it’s a good idea to tell the user what is happening. This is often done with some sort of message on the page saying something like “stuff is loading here, won’t be long”. Also it’s a good idea for the round trip to the server and back to be quick because you don’t want your users to be waiting all day long. However, if the round trip is less than half a second then the user probably didn’t get time to read that message and, depending on the scenario, this could lead to an un-happy web experience.

So I have some code that extends the jQuery AJAX ‘post’ and ‘get’ functions so that we can invoke them but ensure that our success callback function isn’t invoked until at east the N number of milliseconds that we have set have passed.

So the process is as follows:

User clicks a button to request some information.
JavaScript is called and immediately puts some sort of loading message on to the page.
A post or get request is made using my extension and is asked to wait at least 500 milliseconds before the success callback function is invoked.
If the request takes less than 500 milliseconds then the remainder is set to pass before invoking the success function. If the request takes longer than 500 milliseconds then the success function is invoked as soon as the request returns.
The success function removes the loading message from the page and replaces it with the content the user asked for.

This ensures that the user had time to understand the feedback on the page but also ensures the data they requested get’s displayed to them as soon as possible after.

Check out the demo page to see this working.
Download the source to start using it yourself.

Google plus, where to go from here?

Tom Coote — Sat, 02 Jul 2011 15:22:35 +0000

Yesterday I happily got an invite to Google+ from @quagliero. Not sure if it’s still invite only but I’ve been having a go with it yesterday and today. Also installed the Android app which brings everything together really nicely. So far I can safely say I really like the service.

However, I have a bit of a gripe with there being yet another social network service. It’s only a social network if your real world social network are all using the same service!

The problem with saying that not everyone wants service X and that for those people they could use service Y is that being the minority on a social website is useless by contradiction. If the majority of people that you want to connect with socially on the internet use a particular service then your decision is pretty much made for you.

The social web needs to move towards easy integration for it’s users sake. The big guys need to come together and figure out a way of allowing a user of one social network to connect with their friends on a different social network. However, the problem with taking an opensocial approach is that the leading sites end up opening themselves up to losing sign ups and therefore losing potential income.

I guess from a business perspective it comes down to potential income and for that you need sign ups on your social network, not someone else’s. However, I do think that the problem is there to be solved and certainly is solvable. It shouldn’t be open source like opensocial it should be a private network of proven web companies coming together to define a closed approach to integrating their services. Open source isn’t the answer here due to the competitive nature of market share and income.

Aggregators and/or API’s are also not the answer. They force the user to move away from the social network experience they really want due to the equal want to connect with all their friends across different networks.

SimpleModal version 2

Tom Coote — Tue, 14 Jun 2011 16:13:20 +0000

I have just updated my SimpleModal JavaScript component and released it as version 2. You can read about the script and download it from it’s page here.

I’ve just improved it a little to better support newer browsers and tested in the latest jQuery version.

I’d like to mention that I’ve been using this script in all of my web projects for years now. It’s simplicity makes it so flexible which I think is why it’s stood the test of time for me.

Django Class Based Generic API

Tom Coote — Sun, 12 Jun 2011 14:06:47 +0000

It has been a pretty rainy weekend so I’ve decided to do some Python programming and have come up with a set of tools to help build API’s in django.

I know there are a few libraries already available for building API’s in django (I’ve used and contributed to some of them) but since django 1.3 was released with the class based generic views then a lot of the functionality needed to create API’s is already in the framework. So I decided to just build a few extra things to make API’s easy in django by using the framework for most of the heavy lifting.

I’ve made the code available on GitHub. Here are the features in a nut shell;

A new generic class view called APIView which provides a mini framework for supporting different data formats for an API and providing some common API responses.
A throttling decorator to limit API calls per user.
Support for base64 http authentication.

I’ve put basic support in for JSON requests and responses. The code acts as a good basis for a complete API in django and allows an easy way to extend it for new data formats. It makes good use of POST, GET, PUT and DELETE request methods.

View the repository on GitHub. I would be interested to know how anyone gets on using it and if anyone adds in support for different data formats.

Input Character Counter

Tom Coote — Sun, 22 May 2011 16:02:08 +0000

I built this whilst watching Britain’s Got Talent yesterday. I wasn’t enjoying the show to much. It’s a simple jQuery plugin that you can attach to textarea or input elements which will limit the amount of characters that are allowed for the input and also show a counter in the top right corner to indicate the character count.

All you need is the CSS and JavaScript on your page then attach it to something like this;

$(textarea).charCount(100);

Visit the demo page to have a go or download the source.

Daily Windows PostgreSQL Backups

Tom Coote — Thu, 14 Apr 2011 21:12:44 +0000

Recently I installed PostgreSQL on a Windows 2003 server. I also wanted to back up one of the PostgreSQL databases to a folder on the server on a daily basis as an extra precautionary measure. PostgreSQL comes with a program called pg_dump which can be used to create a back up of a database. You give it the database to back up, the file to back up to and it does all the work for you.

The extra work needed here is to back up to a file name that differs each day so that you can have a folder of back ups per day. It would be handy for pg_dump to generate a file name based on date but it doesn’t. So I created a batch file to do the job in one go including a file name based on the date.

@echo off
set date=%date:/=_%
set BACKUP_FILE=D:\PostgreSQL_Backups\DBName\DBName_%date%.backup
pg_dump -h localhost -p 5432 -U postgres -F p -b -v -f %BACKUP_FILE% DBName

Then I created a Windows task scheduled to run daily which just runs the above batch file.

Ajax HTML5 in IE

Tom Coote — Fri, 04 Mar 2011 20:31:25 +0000

We have recently moved towards the use of HTML 5 at my work. For those who have done the same you will already know of the extra work you need to do before HTML 5 will work in Internet Explorer browsers. The way we did it, which is probably the way most people do it, was to add the html5shiv script onto the web page if IE browsers are being used. This script will get you up and running unless you need to AJAX HTML 5 content into the page. That content will not get parsed and so will not behave properly. The answer to that is another script called innershiv which provides a function to pass HTML 5 content to and receive that content back parsed ready for IE browsers.

Both of these scripts are really useful and for most people they are pretty much the difference between making use of HTML 5 or not. The one problem that remains is all the extra work needed to run HTML content through the innershiv function at the end of every single AJAX request in IE browsers. It’s a lot of work! So for those of you who use jQuery I can show you how to tell jQuery to run all content from AJAX requests through innershiv automatically for IE so that you don’t have to do any more coding than you normally would.

This script will work if the browser version is IE8 or lower and relies on jQuery, html5shiv and innershiv all being on the page.

I haven’t tested this code a great deal so far, it’s something I built today. It appears to work very well on the web site I’m currently building. All comments welcome as always!