I knew when I had an opportunity to work an application which handled sensitive information, that it had to be done with the protection of this information as the number one priority.

I began familiarizing myself with the application by starting with the PostgreSQL database back-end. The pgcrypto module was present, but none of it’s functions were currently in use.

For the remainder of the article, we will take a look at how to use some of the pgcrypto PGP encryption functions to encrypt sensitive data.

First we will need to create our table. For the purposes of our example, the table will contain only one column.

CREATE TABLE messages (
   message bytea
);

Next we need to create our PGP keys. This process is outlined in the PostgreSQL pgcrypto documentation here.

Once we’ve created our PGP keys we can test the encryption functionality using the below Python code. Here we are using the public key to encrypt the word “testing” and inserting the result into our table.

#!/usr/bin/env python

import psycopg2

conn = psycopg2.connect("dbname='crypto_test' user='tgc' host='localhost'")

curs = conn.cursor()

f = open("public.key", "r")
key = f.read()

curs.execute("INSERT INTO messages (message) VALUES (pgp_pub_encrypt('testing', dearmor('%s')))" % (key))

conn.commit()

Did it work? Let’s see what was actually inserted into our table.

crypto_test=# select * from messages;
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             message
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 \301\301N\003~_\240V\321\032$\006\020\010\000\271H9\327\224\0336\245:\242\202\352B\232\314\261\2263\032#Z\343\344\365;m\333\337\3226`\352\344\017H\304\373|\213\022\246\343$\351Z\307\366`<\266\206<\330\332\024\203xX\270S\261\317\\\317\336bz\241\307\337\333`\203\020\311J?Y\013\331\021b{\220[S\011\032eu\374s\265l\336JiD\217\364P\275\272\004i\220\214\205\3111\206<\361\025%\235+U\253\202\340\250V\247\361\310\334f\307\322x=>\315x\263;\307w\335\351~--\363\350\361]5)\203\327"\254vk4o\211\244s^r\301\310\374=\312L\347\233\263\232\256~\337\033tM>\012\001C\200\345\303r\326\354{\005\3149\314\211\246\210\273\002~Z\317\270p\237\376\307\025\377r\370\221G\324\306\303\355'\301\020\357/\2049\342\222}\327K\321\347\322+\343Y\220C\027vn\027\356V-\203\355\311\033\355A)\277\266\224\256\022\007\377_\031:\015\345-\264Rj~n\207\240/\370D\3143z\305k\245/f'\004\230r[\207\321H\240a^\247\015\037\351\276!~\354\031\314~)\224\364r\354\036\337\376\006\024K\300ih\335\377\011\256C\262\253\016\210ZN\272\231{\332\304\303\332\003;\221\022\004\275\335\330$\027v\355\004,f\2713\266\352\311\027\326\316\3742LMM\353\037\222C\264\240\275>,\210o\326\212F_@G\306\276\274\332\247\256\336\027U\377\224\313\221>b\355\003K\300\231c<H\236;[\027!\263\311\260V\230\200Jp\333\023\027\004Y\273\321\306N\024%\376\010\317\351\016\332&\030\213\225\017#\006\243nb\332r\363~\013\336\207\337\005\024\322\351$KG\334\376\260\366\025]K\031\037w\016O\2358\277\032?s%\300R\263\031R\273S:\376&\377\326\365\240_\252+;\002+\243\034\257\364\377\322U\320F\005\260\230OS\373~\3228\001*\304\265\204\273\335\236\022\305\312\233\350\011\231\211\250\2103\273\305\214\1770g\305k\254\245!X\262\216\000\020K%~OsJ\365m"\330*\246KX"K4\350\013\214#
(1 row)

Well, that’s certainly a far cry from “testing”.

Now for the code that decrypts our message.

#!/usr/bin/env python

import psycopg2

conn = psycopg2.connect("dbname='crypto_test' user='tgc' host='localhost'")

curs = conn.cursor()

f = open("secret.key", "r")
key = f.read()

curs.execute("SELECT pgp_pub_decrypt(message, dearmor('%s')) from messages" % (key))

rows = curs.fetchall()

for row in rows:
    print row[0]

Let’s try running our decryption script

$ python ./test-decrypt.py
testing

Note: depending on the type of data you are encrypting, the functions pgp_pub_encrypt_bytea and pgp_pub_decrypt_bytea are also available.

<!-- books.xml -->
<catalog>
  <book isbn="1-880985-26-8">
    <title>The Consumer</title>
    <author>M. Gira</author>
  </book>
  <book isbn="0-679775-43-9">
    <title>The Wind-Up Bird Chronicle</title>
    <author>Haruki Murakami</author>
  </book>
  <!-- imagine more entries here... -->
</catalog>

Now we can create the database table.

create table books
(
isbn varchar(14) primary key not null,
title varchar(50),
author varchar(50)
);

Our example depends on SQLAlchemy, which is a “Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.”

While our example leverages a Postgres database, given SQLAlchemy’s database agnostic approach, it can trivially be modified to access a MySQL database.

This code is derived from a SAX parser originally found in Programming Python 3rd Edition.

# bookhandler.py
from sqlalchemy import *
from sqlalchemy.orm import *

import xml.sax.handler

pg_db = create_engine('postgres:///testdb?user=homer')

metadata = MetaData(pg_db)

books_table = Table('books', metadata, autoload=True)

class Book(object):
    pass

mapper(Book, books_table)

class BookHandler(xml.sax.handler.ContentHandler):
    def __init__(self):
        self.buffer = ""
        self.inField = 0
        self.session = create_session(bind=pg_db)

    def startElement(self, name, attributes):
        if name == "book":
            self.isbn = attributes["isbn"]
        elif name == "title":
            self.inField = 1
        elif name == "author":
            self.inField = 1

    def characters(self, data):
        if self.inField:
            self.buffer += data

    def endElement(self, name):
        if name == "book":
            self.session.begin()
            self.newbook = Book()
            self.newbook.isbn = self.isbn
            self.newbook.title = self.title
            self.newbook.author = self.author
            self.session.save(self.newbook)
            self.session.commit()
        elif name == "title":
            self.inField = 0
            self.title = self.buffer
        elif name == "author":
            self.inField = 0
            self.author = self.buffer
        self.buffer = ""

Now that we’ve set up our sax parser and handler to parse and load the entries from books.xml into the table, lets set up a small script to drive it:

# runit.py
import bookhandler
import xml.sax

parser = xml.sax.make_parser()
handler = bookhandler.BookHandler()
parser.setContentHandler(handler)
parser.parse("books.xml")

Now let’s see if it works:

$ ls
bookhandler.py  books.xml       runit.py
$ python ./runit.py
$ psql testdb

testdb=# select * from books;
     isbn      |           title            |     author
---------------+----------------------------+-----------------
 1-880985-26-8 | The Consumer               | M. Gira
 0-679775-43-9 | The Wind-Up Bird Chronicle | Haruki Murakami
(2 rows)

\234\370\260\260\357

This data will need to be escaped as

\\234\\370\\260\\260\\357

in order for Postgres to store it correctly. Fortunately Psycopg2, which we will use in our import script, takes care of the escaping for you. Below is a script that will take an image file (or any binary file) and a user name, stored in the column username and update the column avatar_image (which is of type bytea) with the supplied image file. Make special note of psycopg2.Binary() which wraps the binary representation of the image file, and takes care of the necessary escaping.

#!/usr/bin/env python

import psycopg2
import sys

conn = psycopg2.connect("dbname='myforum' user='johndoe' host='localhost'")

curs = conn.cursor()

f = open(sys.argv[1], 'rb')
binary = f.read()

curs.execute("UPDATE usertable SET avatar_image = %s WHERE username = %s", (psycopg2.Binary(binary), sys.argv[2]))
conn.commit()

This example determines distance between the two points in miles, however it can be modified to return distance in kilometers by replacing 3959.0 (the average radius of the earth in miles) with 6371.0 (the radius in kilometers).

This code does no validation on the existence of the zip code, and will likely return valid coordinates of something (whatever Google thinks you really meant) when you pass it a non existent zip code.

The original geocode code was found here.

Note: You will have to provide your Google Maps API key in the key variable.

import math
import urllib

def get_lat_long(location):
        key = ""
        output = "csv"
        location = urllib.quote_plus(location)
        request = "http://maps.google.com/maps/geo?q=%s&output=%s&key=%s" % (location, output, key)
        data = urllib.urlopen(request).read()
        dlist = data.split(',')
        if dlist[0] == '200':
                return dlist[2], dlist[3]
        else:
                return None, None

def calculate_distance(zip1, zip2):
        lat1, lon1 = get_lat_long(zip1)
        lat2, lon2 = get_lat_long(zip2)

        if (not lat1) or (not lon1) or (not lat2) or (not lon2):
                return -1

        lat1 = float(lat1) * math.pi/180
        lon1 = float(lon1) * math.pi/180
        lat2 = float(lat2) * math.pi/180
        lon2 = float(lon2) * math.pi/180

        return 3959.0 * math.acos(math.sin(lat1) * math.sin(lat2) + math.cos(lat1) * math.cos(lat2) * math.cos(lon2-lon1))