Have you ever looked at the robots.txt file of a web site? They exist so that web sites can stop search engines and other similar robots from indexing web pages on a web site. However they are often used to hide web pages on live web sites that the web site owners don’t want people to see. This is a bad way of using robots.txt but never the less it happens a lot.

What this shows for anyone who feels like taking a look is a list of web page paths under a web site that are marked as disallowed. Those lines marked disallowed are like big red buttons with a sign above them saying “don’t press the big red button”. You’re going to take a look!

So recently Steve asked me to build a quick page that will find the robots.txt file of a web site and automagically create clickable links for all the paths in the file. This makes it really easy to go hunting on web sites for secret pages.

Have a go: robotsneak.com

EDIT: Steve and I have now turned the quick page into a fully designed, emotive, masterpiece

South, the schema and data migration tool for django is really useful. I’ve only started using it myself recently but the benefit is immediate. South creates python files which hold all the information for migrations each time you change the database schema. Each person in a team of developers needs to run these migrations on their development database to keep things up to date and working.

The question is how does one person in a team know that another has changed the schema? If time has passed then simply asking could get a somewhat hazy response.

We use SVN to keep track of code changes amongst the team of developers. SVN is a good place to look to see if migrations were committed. The only thing better than looking in SVN logs for migrations is for SVN to tell us when migrations are committed. So that is what I’ve had a go at doing.

Using the SVN pre-commit hook I’ve been able to send emails to those involved in a project each time someone commits code that contains new database migration changes. The first part of this was to create a pre-commit.bat file in the hooks directory of my projects repository.

C:\python26\python.exe D:\repositories\post_commit_emailer.py %* tom@email.com bob@email.com eddy@email.com

This batch file is run by SVN just before a commit happens. It is passed the repository and transaction name when called. As you can see, I’ve also appended some extra parameters which are the emails of everyone involved with this project.

So the next bit is the python script that this batch file calls.

import smtplib
import sys
import pysvn
import re
from datetime import datetime

SERVER = 'localhost'
FROM = 'svn@email.co.uk'
SUBJECT = 'SVN South Migrations Committed'
TEXT = \
'''
The repository location '%s' was committed and included database schema migrations.

Next time you update your working copy for this repository be sure to run the new south migrations on your local database.

The changed files that are thought to be schema migrations are:
   - %s
'''

try:
    # what was committed?
    repo = sys.argv[1]
    txn = sys.argv[2]
    transaction = pysvn.Transaction(repo, txn)
    changes = transaction.changed()

    # look for database migrations that may have been part of this commit
    migrations = []
    for f in changes.keys():
        if re.search(r'migrations/.*\.py$', f):
            migrations.append(f)

    if migrations:
        # prepare the message. Emails to send to are passed as args
        emails = sys.argv[3:]
        message = TEXT % (repo, '\n   - '.join(migrations))
        message = "From: %s\nTo: %s\nSubject: %s\n\n%s" % \
                    (FROM, ', '.join(emails), SUBJECT, message)

        # send the email
        server = smtplib.SMTP(SERVER)
        server.sendmail(FROM, emails, message)
        server.quit()

except Exception as e:
    log = open('C:\\svn_emailer_log.txt', 'a')
    log.write('\n%s : %s' % (datetime.now().isoformat(), str(e)))
    log.close()

This script uses pysvn to grab all the changes that are to be committed in this transaction. Once it has that information it looks at the path of each file in the list of changes to find any in a migrations folder. South puts migration files in a folder named migrations. If it finds any then it composes an email to everyone in the parameter list. The email lists the migrations that were committed.

So now everyone involved in a project will get an email if someone commits database schema changes. I’m going to start using this for our django projects and see how it goes.

I’ve been using Wingwares python IDE for Django development for almost 2 years. The Wingware IDE has been very good but I have had a few superficial annoyances with it so decided to look around and see what other IDE’s I could use. This led me to Eclipse. The first time I had a look around for Django IDE’s I remember looking at Eclipse but I think I recall difficulties when trying to run Django projects. Eclipse doesn’t natively support Django, you have to install Pydev to get that. After revisiting Eclipse and Pydev I can safely say I’ve switched from Wingware to Eclipse for Django development. The Eclipse IDE in itself is very good and full of useful development features. The Pydev plugin for it which allows python and Django development is also fantastic as it has a GUI for creating and managing Django specific python projects. Another great feature of Eclipse is it’s free price tag where as a single developer license for Wingware is $179 (roughly £115).

To get my Eclipse set up for Django I installed the following:

1. Eclipse Classic 3.6.0
2. Pydev

Then I read through and followed a few guides to understand Pydev and Django support:

1. Pydev getting started guide
2. Pydev Django integration guide

In less than an hour I was up and running and fully understood how to start and develop Django projects in Eclipse. I’d even moved 3 existing Django projects into Eclipse in that time.

Sorry Wingware, you’ve been good to me but I’m trying Eclipse for a while and I don’t think I’m coming back.

I have been using a tool called Sphinx to help write some documentation recently. Writing documentation is boring, time consuming and rubbish. Sphinx doesn’t magically make all those problems go away but it is a great tool none the less.

• It has a nice fluid syntax which is near to how I’d normally jot down notes in notepad.
• It’s layered so that it separates the actual documentation files from the presentation of those files.
• Presentation can be in the form of a complete HTML web site or PDF (amongst others) and this is all done automatically for you.
• It allows for theming of the HTML output so you can integrate you’re documentation directly into your web site.
• Creation of contents tables, index pages and search area is all done, no effort required.

I’ve tried writing documentation both in a word document and as a HTML web page. Both methods are a nightmare to manage with infinite page long word documents and the constant need to write all the HTML tags/links for the web page method. I just want to write the words that make my documentation and that’s what Sphinx lets me do.

Now where can I get a GUI for it?