Sometimes, when developing a web application, you want to acquire data from another source to put on your site. For example, portal sites like portaloo place the latest news headlines on their pages, and the news headlines are constantly updated without human intervention.

I ran into a similar problem designing the web site and database system of Memphis Scholastic Chess. They were spending 5-6 hours per week manually locating the latest ratings for 900 players (having to navigate through around 75 individual pages) and then skim the list for players in our organization on the United States Chess Federation web site. I used Python to write a server site program that runs automatically once a week and downloads and parses pages from the USCF's web site.

Next: The problem..and the solution »