iamtgc » Python

Importing XML into a database with Python and SQLAlchemy

tgc — Tue, 29 Jan 2008 21:19:47 +0000

Let’s begin by analyzing the XML we want to import into our database, it consists of a book’s ISBN, title, and author.
The Consumer M. Gira The Wind-Up Bird Chronicle Haruki Murakami

Now we can create the database table.
create table books ( isbn varchar(14) primary key not null, title varchar(50), author varchar(50) );

Our example depends on SQLAlchemy, which is a “Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.”

While our example leverages a Postgres database, given SQLAlchemy’s database agnostic approach, it can trivially be modified to access a MySQL database.

This code is derived from a SAX parser originally found in Programming Python 3rd Edition.

# bookhandler.py from sqlalchemy import * from sqlalchemy.orm import * import xml.sax.handler pg_db = create_engine('postgres:///testdb?user=homer') metadata = MetaData(pg_db) books_table = Table('books', metadata, autoload=True) class Book(object): pass mapper(Book, books_table) class BookHandler(xml.sax.handler.ContentHandler): def __init__(self): self.buffer = "" self.inField = 0 self.session = create_session(bind=pg_db) def startElement(self, name, attributes): if name == "book": self.isbn = attributes["isbn"] elif name == "title": self.inField = 1 elif name == "author": self.inField = 1 def characters(self, data): if self.inField: self.buffer += data def endElement(self, name): if name == "book": self.session.begin() self.newbook = Book() self.newbook.isbn = self.isbn self.newbook.title = self.title self.newbook.author = self.author self.session.save(self.newbook) self.session.commit() elif name == "title": self.inField = 0 self.title = self.buffer elif name == "author": self.inField = 0 self.author = self.buffer self.buffer = ""
Now that we’ve set up our sax parser and handler to parse and load the entries from books.xml into the table, lets set up a small script to drive it:
# runit.py import bookhandler import xml.sax parser = xml.sax.make_parser() handler = bookhandler.BookHandler() parser.setContentHandler(handler) parser.parse("books.xml")

Now let’s see if it works:
$ ls bookhandler.py books.xml runit.py $ python ./runit.py $ psql testdb testdb=# select * from books; isbn | title | author ---------------+----------------------------+----------------- 1-880985-26-8 | The Consumer | M. Gira 0-679775-43-9 | The Wind-Up Bird Chronicle | Haruki Murakami (2 rows)

Using Python to Load Binary Files into Postgres

tgc — Sat, 04 Aug 2007 18:47:47 +0000

Many times developers may find it desirable to store binary files within a database. For example, an online store may find it easier to store the pictures of their merchandise in the database along side the item description, quantity, price, etc. Doing this can have it’s advantages and disadvantages. Advantages may include easier maintenance, as there are no image files to backup and no need to worry about filename collisions. However the disadvantages of the overhead of database queries, especially if you are dealing with large image files, will have to be weighed.

The primary obstacle in importing binary data into Postgres is making sure that your data is properly escaped. For example when you look at an image file, it may contain the following binary data
\234\370\260\260\357
This data will need to be escaped as
\\234\\370\\260\\260\\357
in order for Postgres to store it correctly. Fortunately Psycopg2, which we will use in our import script, takes care of the escaping for you. Below is a script that will take an image file (or any binary file) and a user name, stored in the column username and update the column avatar_image (which is of type bytea) with the supplied image file. Make special note of psycopg2.Binary() which wraps the binary representation of the image file, and takes care of the necessary escaping.
#!/usr/bin/env python import psycopg2 import sys conn = psycopg2.connect("dbname='myforum' user='johndoe' host='localhost'") curs = conn.cursor() f = open(sys.argv[1], 'rb') binary = f.read() curs.execute("UPDATE usertable SET avatar_image = %s WHERE username = %s", (psycopg2.Binary(binary), sys.argv[2])) conn.commit()

Geocoding with Python

tgc — Wed, 25 Jul 2007 20:57:37 +0000

The following code snippet is intended to demonstrate how you can leverage Python and the Google Maps/Geocode API to query the coordinates of two zip codes and then determine the distance between them. Both functions have far reaching uses independently which will be further demonstrated in future posts.

This example determines distance between the two points in miles, however it can be modified to return distance in kilometers by replacing 3959.0 (the average radius of the earth in miles) with 6371.0 (the radius in kilometers).

This code does no validation on the existence of the zip code, and will likely return valid coordinates of something (whatever Google thinks you really meant) when you pass it a non existent zip code.

The original geocode code was found here.

Note: You will have to provide your Google Maps API key in the key variable.

import math import urllib def get_lat_long(location): key = "" output = "csv" location = urllib.quote_plus(location) request = "http://maps.google.com/maps/geo?q=%s&output=%s&key=%s" % (location, output, key) data = urllib.urlopen(request).read() dlist = data.split(',') if dlist[0] == '200': return dlist[2], dlist[3] else: return None, None def calculate_distance(zip1, zip2): lat1, lon1 = get_lat_long(zip1) lat2, lon2 = get_lat_long(zip2) if (not lat1) or (not lon1) or (not lat2) or (not lon2): return -1 lat1 = float(lat1) * math.pi/180 lon1 = float(lon1) * math.pi/180 lat2 = float(lat2) * math.pi/180 lon2 = float(lon2) * math.pi/180 return 3959.0 * math.acos(math.sin(lat1) * math.sin(lat2) + math.cos(lat1) * math.cos(lat2) * math.cos(lon2-lon1))