iamtgc » Postgres

PostgreSQL Database Encryption

tgc — Tue, 13 Sep 2011 14:01:49 +0000

For last decade data breaches and data losses of Social Security numbers, credit card numbers, or one of a slew of other pieces of private information, have landed businesses on the front page of major newspapers, and in some instances with legal liability.

I knew when I had an opportunity to work an application which handled sensitive information, that it had to be done with the protection of this information as the number one priority.

I began familiarizing myself with the application by starting with the PostgreSQL database back-end. The pgcrypto module was present, but none of it’s functions were currently in use.

For the remainder of the article, we will take a look at how to use some of the pgcrypto PGP encryption functions to encrypt sensitive data.

First we will need to create our table. For the purposes of our example, the table will contain only one column.

CREATE TABLE messages (
   message bytea
);

Next we need to create our PGP keys. This process is outlined in the PostgreSQL pgcrypto documentation here.

Once we’ve created our PGP keys we can test the encryption functionality using the below Python code. Here we are using the public key to encrypt the word “testing” and inserting the result into our table.

#!/usr/bin/env python

import psycopg2

conn = psycopg2.connect("dbname='crypto_test' user='tgc' host='localhost'")

curs = conn.cursor()

f = open("public.key", "r")
key = f.read()

curs.execute("INSERT INTO messages (message) VALUES (pgp_pub_encrypt('testing', dearmor('%s')))" % (key))

conn.commit()

Did it work? Let’s see what was actually inserted into our table.

crypto_test=# select * from messages;
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             message
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 \301\301N\003~_\240V\321\032$\006\020\010\000\271H9\327\224\0336\245:\242\202\352B\232\314\261\2263\032#Z\343\344\365;m\333\337\3226`\352\344\017H\304\373|\213\022\246\343$\351Z\307\366`<\266\206<\330\332\024\203xX\270S\261\317\\\317\336bz\241\307\337\333`\203\020\311J?Y\013\331\021b{\220[S\011\032eu\374s\265l\336JiD\217\364P\275\272\004i\220\214\205\3111\206<\361\025%\235+U\253\202\340\250V\247\361\310\334f\307\322x=>\315x\263;\307w\335\351~--\363\350\361]5)\203\327"\254vk4o\211\244s^r\301\310\374=\312L\347\233\263\232\256~\337\033tM>\012\001C\200\345\303r\326\354{\005\3149\314\211\246\210\273\002~Z\317\270p\237\376\307\025\377r\370\221G\324\306\303\355'\301\020\357/\2049\342\222}\327K\321\347\322+\343Y\220C\027vn\027\356V-\203\355\311\033\355A)\277\266\224\256\022\007\377_\031:\015\345-\264Rj~n\207\240/\370D\3143z\305k\245/f'\004\230r[\207\321H\240a^\247\015\037\351\276!~\354\031\314~)\224\364r\354\036\337\376\006\024K\300ih\335\377\011\256C\262\253\016\210ZN\272\231{\332\304\303\332\003;\221\022\004\275\335\330$\027v\355\004,f\2713\266\352\311\027\326\316\3742LMM\353\037\222C\264\240\275>,\210o\326\212F_@G\306\276\274\332\247\256\336\027U\377\224\313\221>b\355\003K\300\231c
Well, that’s certainly a far cry from “testing”.
Now for the code that decrypts our message.
#!/usr/bin/env python

import psycopg2

conn = psycopg2.connect("dbname='crypto_test' user='tgc' host='localhost'")

curs = conn.cursor()

f = open("secret.key", "r")
key = f.read()

curs.execute("SELECT pgp_pub_decrypt(message, dearmor('%s')) from messages" % (key))

rows = curs.fetchall()

for row in rows:
    print row[0]

Let’s try running our decryption script
$ python ./test-decrypt.py
testing

Note: depending on the type of data you are encrypting, the functions pgp_pub_encrypt_bytea and pgp_pub_decrypt_bytea are also available.

Exploring Triggers in MySQL and PostgreSQL

tgc — Mon, 15 Aug 2011 13:25:37 +0000

Recently I was working on a car maintenance database. The database stores, among other things, odometer reading, miles per fill up, number of gallons per fill up, and cost per gallon. One of the missing pieces of information I wanted to report on is the miles per gallon I was achieving. While this is trivial to insert, I didn’t want to modify the client which was already deployed and running. Since the data to determine this value was already present, I turned to database triggers to help me record this value.

Wikipedia describes “A database trigger is procedural code that is automatically executed in response to certain events on a particular table or view in a database.”

In our case, the procedural code we want to execute is the computation of our miles per gallon, and the event we are responding to is an insert into our table. In this article I will outline how to create these triggers in both MySQL and PostgreSQL.

Our triggers will be applied to the following table.

CREATE TABLE mileage (
   miles float,
   gallons float,
   mpg float
);

The following was tested in MySQL 5.5.14, but should work on other versions as well.

Let’s create our trigger. The key things to note here are BEFORE INSERT, since we want to insert the mpg value along with the miles and gallons, and NEW, since we are referring to a column of a new row to be inserted.

-- MySQL
CREATE TRIGGER trig__mileage BEFORE INSERT ON mileage FOR EACH ROW SET NEW.mpg  = NEW.miles / NEW.gallons;

Now with our trigger created, we can test to see if it fires correctly. If it fires correctly, we can expect our mpg column to be populated, otherwise it will be NULL.

-- MySQL
INSERT INTO mileage (miles, gallons) VALUES ( 250, 10 )

SELECT * FROM mileage;
+-------+---------+------+
| miles | gallons | mpg  |
+-------+---------+------+
|   250 |      10 |   25 |
+-------+---------+------+

Success! It really is that simple. Additional documentation, including more complex examples, can be found here.

Now on to PostgreSQL. These were tested on PostgreSQL 8.3.15, but should work on other versions as well.

The first thing to note is the CREATE TRIGGER statement in PostgreSQL takes a user supplied function as an argument. This function must return type TRIGGER, and in our instance will be responsible for calculating our miles per gallon.
In our function we use the NEW variable, which holds the new database row for insert.

-- PostgreSQL
CREATE OR REPLACE FUNCTION trig_proc__calculate_mpg() RETURNS TRIGGER AS $_$
BEGIN
   NEW.mpg := NEW.miles / NEW.gallons;
   RETURN NEW;
END $_$
LANGUAGE plpgsql;

With our function created, we now create the trigger.

-- PostgreSQL
CREATE TRIGGER trig__mileage BEFORE INSERT ON mileage FOR EACH ROW EXECUTE PROCEDURE trig_proc__calculate_mpg();

And now to test if our trigger fires correctly.

-- PostgreSQL
INSERT INTO mileage (miles, gallons) VALUES ( 312.3, 12.1 );

SELECT * FROM mileage;
 miles | gallons |       mpg
-------+---------+------------------
 312.3 |    12.1 | 25.8099173553719

As you can see, our trigger has populated the mpg column. Additional documentation on triggers in Postgres can be found here and here.

Returning Composite Types in Postgres Stored Procedures

tgc — Thu, 01 Oct 2009 14:52:38 +0000

Expanding on the PostgreSQL examples in our previous post, here we will look at taking advantage of some of the features in PostgreSQL 8.3 and modifying our zip_proximity function to avoid using cursors and instead define and return a set of our own composite type. From the postgres documentation “A composite type describes the structure of a row or record…”. Since we know that our stored procedure will return a row including: zip code, latitude, longitude, city, state, state abbreviation, and distance, we can create a composite type called, for example, ziprowtype.

Here we will create the composite type.

CREATE type ziprowtype as (zip varchar, lat float, lon float, city varchar, state varchar, state_abbrev varchar, distance float);

Now, to modify the stored procedure, you will need to change the return type from refcursor to SETOF ziprowtype. The body of the function changes a bit too.
First we load the result of our query into record type “r”, then loop and “RETURN NEXT r”.

CREATE OR REPLACE FUNCTION zip_proximity2(varchar, double precision, varchar) RETURNS SETOF ziprowtype
    AS $_$
   DECLARE
      home_lat float;
      home_lon float;
      r record;
   BEGIN
      SELECT lat, lon INTO home_lat, home_lon FROM zipcodes WHERE zip = $1;
      FOR r IN
      SELECT zip, lat, lon, city, state, state_abbrev, calculate_distance($3, home_lat, home_lon, lat, lon) AS distance
          FROM zipcodes WHERE calculate_distance($3, home_lat, home_lon, lat, lon) < $2 ORDER BY distance
      LOOP
         RETURN NEXT r;
      END LOOP;
   END;
   $_$
    LANGUAGE plpgsql;

NOTE: To see how we implemented calculate_distance, please read this post.

Now, instead of using cursors and transactions, we can use the following query to return the desired results.

testdb=# select * from zip_proximity2('94043', 3.0, 'mi');
  zip  |   lat    |    lon     |     city      |   state    | state_abbrev |     distance
-------+----------+------------+---------------+------------+--------------+-------------------
 94043 | 37.42337 | -122.07981 | MOUNTAIN VIEW | CALIFORNIA | CA           |                 0
 94039 | 37.41884 | -122.09124 | MOUNTAIN VIEW | CALIFORNIA | CA           | 0.701004112864842
 94035 | 37.41753 | -122.05283 | MOUNTAIN VIEW | CALIFORNIA | CA           |  1.53459076775345
 94042 | 37.39314 | -122.07827 | MOUNTAIN VIEW | CALIFORNIA | CA           |  2.09052870375317
 94041 | 37.38961 | -122.07715 | MOUNTAIN VIEW | CALIFORNIA | CA           |  2.33729808540454
 94306 | 37.41478 | -122.12139 | PALO ALTO     | CALIFORNIA | CA           |  2.35776644118448
 94303 | 37.44424 | -122.11736 | PALO ALTO     | CALIFORNIA | CA           |  2.51480871338068
(7 rows)

NOTE: If you're getting ERROR: wrong record type supplied in RETURN NEXT, then it's likely your composite type does not match the columns you're querying.

You can also select any subset of the row (composite type) using standard SQL.

testdb=# select zip, distance from zip_proximity2('94043', 3.0, 'mi') limit 5;
  zip  |     distance
-------+-------------------
 94043 |                 0
 94039 | 0.701004115082249
 94035 |  1.53459076784732
 94042 |  2.09052870372395
 94041 |  2.33729808557031
(5 rows)

As always, please feel free to leave a comment with any questions or suggestions.

Implementing Zip Code Proximity Functions in MySQL and PostgreSQL

tgc — Wed, 14 Jan 2009 17:26:08 +0000

On most retail and social networking websites (along with many others), you’ll have the capability to search for people, businesses, store locations, etc within a given distance of your location. This can be implemented in a number of ways both mathematically and programmatically. In an attempt to reduce the amount of code I (or others) have to write, be it in PHP, Python, or any number languages that may interface with our database, I have chosen to implement these zip code proximity and distance functions as stored procedures in the database.

To start, you may already have your database with Zip Code coordinates, if you are looking for one, Team RedLine offers an excellent Zip Code Database for $5 US that can be easily imported into the database of your choice.

The table that we will be using was created as follows:

CREATE TABLE zipcodes (
    zip varchar(5),
    lat double precision,
    lon double precision,
    city varchar(30),
    state varchar(30),
    state_abbrev varchar(2));

I use 57.2958 as a constant for 180 / π to convert between degrees and radians, beyond this I will focus on the implementation and point you to the Haversine Formula and the Law of Consines if you wish to read more on the math.

First we will examine how these functions are written for PostgreSQL. (MySQL functions are further down)

The first function implements the Law of Consines, and allows the user to specify which measurement (miles or kilometers) to use, to determine the distance between two coordinates.

CREATE FUNCTION calculate_distance(varchar, double precision, double precision, double precision, double precision) RETURNS double precision
    AS $_$
   DECLARE
      earth_radius double precision;
   BEGIN
      IF $1 = 'mi' THEN
         earth_radius := 3959.0;
      ELSIF $1 = 'km' THEN
         earth_radius := 6371.0;
      END IF;
      RETURN earth_radius * acos(sin($2 / 57.2958) * sin($4 / 57.2958) + cos($2/ 57.2958) * cos($4 / 57.2958) * cos(($5 / 57.2958) - ($3 / 57.2958)));
   END;
   $_$
    LANGUAGE plpgsql;

The calculate_distance function can certainly be used stand alone, but in our example it is called exclusively from our zip_proximity function below.

zip_proximity takes a refcursor, a zipcode, a distance, and a distance metric (‘mi’ or ‘km’).
Update: Please see this article on implementing this function without using refcursor.

First we retrieve the coordinates for the “home” zipcode, and query the databases, performing calculate_distance on each entry in the database, capturing only those that fall withing the given distance. We add a field to the cursor we return, which is the distance from the home zipcode, to the zipcode which fell within our provided distance.

CREATE FUNCTION zip_proximity(refcursor, character, double precision, varchar) RETURNS refcursor
    AS $_$
   DECLARE
      home_lat float;
      home_lon float;
   BEGIN
      SELECT lat, lon INTO home_lat, home_lon FROM zipcodes WHERE zip = $2;
      OPEN $1 FOR
         SELECT zip, lat, lon, city, state, state_abbrev, calculate_distance($4, home_lat, home_lon, lat, lon) AS distance FROM zipcodes
             WHERE calculate_distance($4, home_lat, home_lon, lat, lon) < $3 ORDER BY distance;
      RETURN $1;
   END;
   $_$
    LANGUAGE plpgsql;

Now we will query all zipcodes within a 3 mile radius of Mountain View, California 94043.
Here is how we call our new function(s). Since we are using cursors, we need to be in a transaction.

testdb=# BEGIN;
BEGIN
testdb=# SELECT zip_proximity('zc', '94043', 3, 'mi');
 zip_proximity
---------------
 zc
(1 row)

testdb=# FETCH ALL FROM zc;
  zip  |   lat    |    lon     |     city      |   state    | state_abbrev |     distance
-------+----------+------------+---------------+------------+--------------+-------------------
 94043 | 37.42337 | -122.07981 | MOUNTAIN VIEW | CALIFORNIA | CA           |                 0
 94039 | 37.41884 | -122.09124 | MOUNTAIN VIEW | CALIFORNIA | CA           | 0.701004112864842
 94035 | 37.41753 | -122.05283 | MOUNTAIN VIEW | CALIFORNIA | CA           |  1.53459076775345
 94042 | 37.39314 | -122.07827 | MOUNTAIN VIEW | CALIFORNIA | CA           |  2.09052870375317
 94041 | 37.38961 | -122.07715 | MOUNTAIN VIEW | CALIFORNIA | CA           |  2.33729808540454
 94306 | 37.41478 | -122.12139 | PALO ALTO     | CALIFORNIA | CA           |  2.35776644118448
 94303 | 37.44424 | -122.11736 | PALO ALTO     | CALIFORNIA | CA           |  2.51480871338068
(7 rows)

testdb=# END;
COMMIT

Now let's take a look at the MySQL functions. The calculate_distance function is essentially identical to the PostgreSQL function above.

DELIMITER //
CREATE FUNCTION calculate_distance(measurement varchar(2), base_lat double precision, base_lon double precision, lat double precision, lon double precision) RETURNS double precision
   BEGIN
      DECLARE earth_radius double precision;
      IF measurement = 'km' THEN
         SET earth_radius = 6371.0;
      ELSEIF measurement = 'mi' THEN
         SET earth_radius = 3959.0;
      END IF;
      RETURN earth_radius * ACOS(SIN(base_lat / 57.2958) * SIN(lat / 57.2958) + COS(base_lat / 57.2958) * COS(lat / 57.2958) * COS((lon / 57.2958) - (base_lon / 57.2958)));
   END //
DELIMITER ;

This function differs slightly from it's Postgres counterpart. Since MySQL does not currently support functions that return a cursor, we will create a procedure which will execute the same query that we would have returned in the cursor.

DELIMITER //
CREATE PROCEDURE zip_proximity(zipcode varchar(5), radius double precision, measurement varchar(2))
   BEGIN
   DECLARE base_lat double precision;
   DECLARE base_lon double precision;
   SELECT lat, lon INTO base_lat, base_lon FROM zipcodes WHERE zip = zipcode;
   SELECT zip, lat, lon, city, state, state_abbrev, calculate_distance(measurement, base_lat, base_lon, lat, lon) AS distance FROM zipcodes
      WHERE calculate_distance(measurement, base_lat, base_lon, lat, lon) < radius ORDER BY distance;
   END //
DELIMITER ;

Now here is how we would call the procedure, again we are querying for all zip codes within a three mile radius of 94043.

mysql> call zip_proximity('94043', 3, 'mi');
+-------+----------+------------+---------------+------------+--------------+-------------------+
| zip   | lat      | lon        | city          | state      | state_abbrev | distance          |
+-------+----------+------------+---------------+------------+--------------+-------------------+
| 94043 | 37.42337 | -122.07981 | MOUNTAIN VIEW | CALIFORNIA | CA           |                 0 |
| 94039 | 37.41884 | -122.09124 | MOUNTAIN VIEW | CALIFORNIA | CA           | 0.701004115082249 |
| 94035 | 37.41753 | -122.05283 | MOUNTAIN VIEW | CALIFORNIA | CA           |  1.53459076784732 |
| 94042 | 37.39314 | -122.07827 | MOUNTAIN VIEW | CALIFORNIA | CA           |  2.09052870372395 |
| 94041 | 37.38961 | -122.07715 | MOUNTAIN VIEW | CALIFORNIA | CA           |  2.33729808557031 |
| 94306 | 37.41478 | -122.12139 | PALO ALTO     | CALIFORNIA | CA           |  2.35776644108718 |
| 94303 | 37.44424 | -122.11736 | PALO ALTO     | CALIFORNIA | CA           |  2.51480871282271 |
+-------+----------+------------+---------------+------------+--------------+-------------------+
7 rows in set (0.75 sec)

Query OK, 0 rows affected (0.75 sec)

Feel free to leave a comment with any questions or suggestions.

Exploring Stored Procedures in MySQL and PostgreSQL

tgc — Fri, 19 Dec 2008 14:38:57 +0000

The PostgreSQL PL/pgSQL procedural language is well documented here and the MySQL Reference Manual is available here. The MySQL documentation is, in my opinion, lacking, however the MySQL Stored Procedure Forum helps a great deal in making up for the lack of documentation.

Now, let’s take for example this simplified user table. In reality it would contain additional information, a hashed password, real name, who knows, but for the sake of our example, we’ll keep it minimal. Once the table is created, we will want to create a stored procedure that automatically expires accounts that have not logged on in more than a month.

CREATE TABLE users (
   username varchar(12),
   last_login timestamp,
   expired boolean
);

Now, we could create a stored procedure that expires all accounts that have not logged on in a month, but to make it more versatile, and to demonstrate more of the capabilities, we will allow the user to define the number of days since last logged on before the account expires.

First, let’s see how this we could write this for PostgreSQL, we will be using the PL/pgSQL procedural language.

CREATE OR REPLACE FUNCTION mark_expired(days INT) RETURNS VOID AS
$$
DECLARE
   delta BIGINT;
BEGIN
   delta := $1*86400;
   UPDATE users SET expired=true WHERE EXTRACT(epoch from age(CURRENT_TIMESTAMP, last_login)) > delta;
END;
$$
LANGUAGE plpgsql;

Let’s look at the user table before running the stored procedure

 username  |     last_login      | expired
----------+---------------------+---------
 homer    | 2008-12-18 00:00:00 | f
 marge    | 2008-11-18 00:00:00 | f
 bart     | 2008-12-14 00:00:00 | f
 lisa     | 2008-12-17 00:00:00 | f
 maggie   | 2008-12-19 00:00:00 | f

Here is how you would execute the stored procedure, with our user defined 30 day argument.

testdb=# select mark_expired(30);
testdb=# select * from users;
 username |     last_login      | expired
----------+---------------------+---------
 homer    | 2008-12-18 00:00:00 | f
 bart     | 2008-12-14 00:00:00 | f
 lisa     | 2008-12-17 00:00:00 | f
 maggie   | 2008-12-19 00:00:00 | f
 marge    | 2008-11-18 00:00:00 | t

Success!, Marge’s account has now been marked expired, since she has not logged on in the last 30 days.

Your first inclination may be to extract the day from age, but this does not work as one may think. Take for example the following…

testdb=# select age(now(), CURRENT_DATE-31);
             age
-----------------------------
 1 mon 1 day 13:02:30.065623
(1 row)

Now if we extract day, this is what we get…

testdb=# select extract(day from age(now(), CURRENT_DATE-31));
 date_part
-----------
         1
(1 row)

This is not in fact what we were looking for, if we were to use this method to expire accounts, this account appears to have been logged in as recently as one day ago, when in fact 31 days have elapsed. So this is why we chose to use epoch, which in this case will get total elapsed seconds.

Now, let’s review how you might write this for MySQL.

DELIMITER //
CREATE PROCEDURE mark_expired (days INT)
BEGIN
   UPDATE users SET expired = true
              WHERE last_login < SUBDATE(CURRENT_TIMESTAMP, INTERVAL days DAY);
END //
DELIMITER ;

In this example we take a slightly different approach, we subtract the days argument from the current timestamp, and see if the last login timestamp is older than this. This leverages MySQL's SUBDATE function and avoids having to convert days into seconds.

Let's take a look at our table again, before calling the procedure

mysql> select * from users;
+----------+---------------------+---------+
| username | last_login          | expired |
+----------+---------------------+---------+
| homer    | 2008-12-18 00:00:00 |       0 |
| marge    | 2008-11-18 00:00:00 |       0 |
| bart     | 2008-12-14 00:00:00 |       0 |
| lisa     | 2008-12-17 00:00:00 |       0 |
| maggie   | 2008-12-19 00:00:00 |       0 |
+----------+---------------------+---------+

And how you would call the procedure in MySQL.

mysql> CALL mark_expired(30);
mysql> select * from users;
+----------+---------------------+---------+
| username | last_login          | expired |
+----------+---------------------+---------+
| homer    | 2008-12-18 00:00:00 |       0 |
| marge    | 2008-11-18 00:00:00 |       1 |
| bart     | 2008-12-14 00:00:00 |       0 |
| lisa     | 2008-12-17 00:00:00 |       0 |
| maggie   | 2008-12-19 00:00:00 |       0 |
+----------+---------------------+---------+

Again, success, Marge's account has been expired.