iamtgc » Postgres

Returning Composite Types in Postgres Stored Procedures

tgc — Thu, 01 Oct 2009 14:52:38 +0000

Expanding on the PostgreSQL examples in our previous post, here we will look at taking advantage of some of the features in PostgreSQL 8.3 and modifying our zip_proximity function to avoid using cursors and instead define and return a set of our own composite type. From the postgres documentation “A composite type describes the structure of a row or record…”. Since we know that our stored procedure will return a row including: zip code, latitude, longitude, city, state, state abbreviation, and distance, we can create a composite type called, for example, ziprowtype.

Here we will create the composite type.
testdb=# create type ziprowtype as (zip varchar, lat float, lon float, city varchar, state varchar, state_abbrev varchar, distance float); CREATE TYPE

Now, to modify the stored procedure, you will need to change the return type from refcursor to SETOF ziprowtype. The body of the function changes a bit too.
First we load the result of our query into record type “r”, then loop and “RETURN NEXT r”.
testdb=# CREATE OR REPLACE FUNCTION zip_proximity2(varchar, double precision, varchar) RETURNS SETOF ziprowtype AS $_$ DECLARE home_lat float; home_lon float; r record; BEGIN SELECT lat, lon INTO home_lat, home_lon FROM zipcodes WHERE zip = $1; FOR r IN SELECT zip, lat, lon, city, state, state_abbrev, calculate_distance($3, home_lat, home_lon, lat, lon) AS distance FROM zipcodes WHERE calculate_distance($3, home_lat, home_lon, lat, lon) < $2 ORDER BY distance LOOP RETURN NEXT r; END LOOP; END; $_$ LANGUAGE plpgsql;
NOTE: To see how we implemented calculate_distance, please read this post.

Now, instead of using cursors and transactions, we can use the following query to return the desired results.
testdb=# select * from zip_proximity2('94043', 3.0, 'mi'); zip | lat | lon | city | state | state_abbrev | distance -------+----------+------------+---------------+------------+--------------+------------------- 94043 | 37.42337 | -122.07981 | MOUNTAIN VIEW | CALIFORNIA | CA | 0 94039 | 37.41884 | -122.09124 | MOUNTAIN VIEW | CALIFORNIA | CA | 0.701004112864842 94035 | 37.41753 | -122.05283 | MOUNTAIN VIEW | CALIFORNIA | CA | 1.53459076775345 94042 | 37.39314 | -122.07827 | MOUNTAIN VIEW | CALIFORNIA | CA | 2.09052870375317 94041 | 37.38961 | -122.07715 | MOUNTAIN VIEW | CALIFORNIA | CA | 2.33729808540454 94306 | 37.41478 | -122.12139 | PALO ALTO | CALIFORNIA | CA | 2.35776644118448 94303 | 37.44424 | -122.11736 | PALO ALTO | CALIFORNIA | CA | 2.51480871338068 (7 rows)
NOTE: If you’re getting ERROR: wrong record type supplied in RETURN NEXT, then it’s likely your composite type does not match the columns you’re querying.

You can also select any subset of the row (composite type) using standard SQL.
testdb=# select zip, distance from zip_proximity2('94043', 3.0, 'mi') limit 5; zip | distance -------+------------------- 94043 | 0 94039 | 0.701004115082249 94035 | 1.53459076784732 94042 | 2.09052870372395 94041 | 2.33729808557031 (5 rows)

As always, please feel free to leave a comment with any questions or suggestions.

Implementing Zip Code Proximity Functions in MySQL and PostgreSQL

tgc — Wed, 14 Jan 2009 17:26:08 +0000

On most retail and social networking websites (along with many others), you’ll have the capability to search for people, businesses, store locations, etc within a given distance of your location. This can be implemented in a number of ways both mathematically and programmatically. In an attempt to reduce the amount of code I (or others) have to write, be it in PHP, Python, or any number languages that may interface with our database, I have chosen to implement these zip code proximity and distance functions as stored procedures in the database.

To start, you may already have your database with Zip Code coordinates, if you are looking for one, Team RedLine offers an excellent Zip Code Database for $5 US that can be easily imported into the database of your choice.

The table that we will be using was created as follows:
CREATE TABLE zipcodes ( zip varchar(5), lat double precision, lon double precision, city varchar(30), state varchar(30), state_abbrev varchar(2));

I use 57.2958 as a constant for 180 / π to convert between degrees and radians, beyond this I will focus on the implementation and point you to the Haversine Formula and the Law of Consines if you wish to read more on the math.

First we will examine how these functions are written for PostgreSQL. (MySQL functions are further down)

The first function implements the Law of Consines, and allows the user to specify which measurement (miles or kilometers) to use, to determine the distance between two coordinates.
CREATE FUNCTION calculate_distance(varchar, double precision, double precision, double precision, double precision) RETURNS double precision AS $_$ DECLARE earth_radius double precision; BEGIN IF $1 = 'mi' THEN earth_radius := 3959.0; ELSIF $1 = 'km' THEN earth_radius := 6371.0; END IF; RETURN earth_radius * acos(sin($2 / 57.2958) * sin($4 / 57.2958) + cos($2/ 57.2958) * cos($4 / 57.2958) * cos(($5 / 57.2958) - ($3 / 57.2958))); END; $_$ LANGUAGE plpgsql;
The calculate_distance function can certainly be used stand alone, but in our example it is called exclusively from our zip_proximity function below.

zip_proximity takes a refcursor, a zipcode, a distance, and a distance metric (‘mi’ or ‘km’).

First we retrieve the coordinates for the “home” zipcode, and query the databases, performing calculate_distance on each entry in the database, capturing only those that fall withing the given distance. We add a field to the cursor we return, which is the distance from the home zipcode, to the zipcode which fell within our provided distance.

CREATE FUNCTION zip_proximity(refcursor, character, double precision, varchar) RETURNS refcursor AS $_$ DECLARE home_lat float; home_lon float; BEGIN SELECT lat, lon INTO home_lat, home_lon FROM zipcodes WHERE zip = $2; OPEN $1 FOR SELECT zip, lat, lon, city, state, state_abbrev, calculate_distance($4, home_lat, home_lon, lat, lon) AS distance FROM zipcodes WHERE calculate_distance($4, home_lat, home_lon, lat, lon) < $3 ORDER BY distance; RETURN $1; END; $_$ LANGUAGE plpgsql;

Now we will query all zipcodes within a 3 mile radius of Mountain View, California 94043.
Here is how we call our new function(s). Since we are using cursors, we need to be in a transaction.
testdb=# BEGIN; BEGIN testdb=# SELECT zip_proximity('zc', '94043', 3, 'mi'); zip_proximity --------------- zc (1 row) testdb=# FETCH ALL FROM zc; zip | lat | lon | city | state | state_abbrev | distance -------+----------+------------+---------------+------------+--------------+------------------- 94043 | 37.42337 | -122.07981 | MOUNTAIN VIEW | CALIFORNIA | CA | 0 94039 | 37.41884 | -122.09124 | MOUNTAIN VIEW | CALIFORNIA | CA | 0.701004112864842 94035 | 37.41753 | -122.05283 | MOUNTAIN VIEW | CALIFORNIA | CA | 1.53459076775345 94042 | 37.39314 | -122.07827 | MOUNTAIN VIEW | CALIFORNIA | CA | 2.09052870375317 94041 | 37.38961 | -122.07715 | MOUNTAIN VIEW | CALIFORNIA | CA | 2.33729808540454 94306 | 37.41478 | -122.12139 | PALO ALTO | CALIFORNIA | CA | 2.35776644118448 94303 | 37.44424 | -122.11736 | PALO ALTO | CALIFORNIA | CA | 2.51480871338068 (7 rows) testdb=# END; COMMIT

Now let’s take a look at the MySQL functions. The calculate_distance function is essentially identical to the PostgreSQL function above.
DELIMITER // CREATE FUNCTION calculate_distance(measurement varchar(2), base_lat double precision, base_lon double precision, lat double precision, lon double precision) RETURNS double precision BEGIN DECLARE earth_radius double precision; IF measurement = 'km' THEN SET earth_radius = 6371.0; ELSEIF measurement = 'mi' THEN SET earth_radius = 3959.0; END IF; RETURN earth_radius * ACOS(SIN(base_lat / 57.2958) * SIN(lat / 57.2958) + COS(base_lat / 57.2958) * COS(lat / 57.2958) * COS((lon / 57.2958) - (base_lon / 57.2958))); END // DELIMITER ;

This function differs slightly from it’s Postgres counterpart. Since MySQL does not currently support functions that return a cursor, we will create a procedure which will execute the same query that we would have returned in the cursor.
DELIMITER // CREATE PROCEDURE zip_proximity(zipcode varchar(5), radius double precision, measurement varchar(2)) BEGIN DECLARE base_lat double precision; DECLARE base_lon double precision; SELECT lat, lon INTO base_lat, base_lon FROM zipcodes WHERE zip = zipcode; SELECT zip, lat, lon, city, state, state_abbrev, calculate_distance(measurement, base_lat, base_lon, lat, lon) AS distance FROM zipcodes WHERE calculate_distance(measurement, base_lat, base_lon, lat, lon) < radius ORDER BY distance; END // DELIMITER ;

Now here is how we would call the procedure, again we are querying for all zip codes within a three mile radius of 94043.
mysql> call zip_proximity('94043', 3, 'mi'); +-------+----------+------------+---------------+------------+--------------+-------------------+ | zip | lat | lon | city | state | state_abbrev | distance | +-------+----------+------------+---------------+------------+--------------+-------------------+ | 94043 | 37.42337 | -122.07981 | MOUNTAIN VIEW | CALIFORNIA | CA | 0 | | 94039 | 37.41884 | -122.09124 | MOUNTAIN VIEW | CALIFORNIA | CA | 0.701004115082249 | | 94035 | 37.41753 | -122.05283 | MOUNTAIN VIEW | CALIFORNIA | CA | 1.53459076784732 | | 94042 | 37.39314 | -122.07827 | MOUNTAIN VIEW | CALIFORNIA | CA | 2.09052870372395 | | 94041 | 37.38961 | -122.07715 | MOUNTAIN VIEW | CALIFORNIA | CA | 2.33729808557031 | | 94306 | 37.41478 | -122.12139 | PALO ALTO | CALIFORNIA | CA | 2.35776644108718 | | 94303 | 37.44424 | -122.11736 | PALO ALTO | CALIFORNIA | CA | 2.51480871282271 | +-------+----------+------------+---------------+------------+--------------+-------------------+ 7 rows in set (0.75 sec) Query OK, 0 rows affected (0.75 sec)

Feel free to leave a comment with any questions or suggestions.

Exploring Stored Procedures in MySQL and PostgreSQL

tgc — Fri, 19 Dec 2008 14:38:57 +0000

The PostgreSQL PL/pgSQL procedural language is well documented here and the MySQL Reference Manual is available here. The MySQL documentation is, in my opinion, lacking, however the MySQL Stored Procedure Forum helps a great deal in making up for the lack of documentation.

Now, let’s take for example this simplified user table. In reality it would contain additional information, a hashed password, real name, who knows, but for the sake of our example, we’ll keep it minimal. Once the table is created, we will want to create a stored procedure that automatically expires accounts that have not logged on in more than a month.
CREATE TABLE users ( username varchar(12), last_login timestamp, expired boolean );

Now, we could create a stored procedure that expires all accounts that have not logged on in a month, but to make it more versatile, and to demonstrate more of the capabilities, we will allow the user to define the number of days since last logged on before the account expires.

First, let’s see how this we could write this for PostgreSQL, will will be using PL/pgSQL procedural language.
CREATE OR REPLACE FUNCTION mark_expired(days INT) RETURNS VOID AS $$ DECLARE delta BIGINT; BEGIN delta := $1*86400; UPDATE users SET expired=true WHERE EXTRACT(epoch from age(CURRENT_TIMESTAMP, last_login)) > delta; END; $$ LANGUAGE plpgsql;

Let’s look at the user table before running the stored procedure
username | last_login | expired ----------+---------------------+--------- homer | 2008-12-18 00:00:00 | f marge | 2008-11-18 00:00:00 | f bart | 2008-12-14 00:00:00 | f lisa | 2008-12-17 00:00:00 | f maggie | 2008-12-19 00:00:00 | f

Here is how you would execute the stored procedure, with our user defined 30 day argument.
testdb=# select mark_expired(30); testdb=# select * from users; username | last_login | expired ----------+---------------------+--------- homer | 2008-12-18 00:00:00 | f bart | 2008-12-14 00:00:00 | f lisa | 2008-12-17 00:00:00 | f maggie | 2008-12-19 00:00:00 | f marge | 2008-11-18 00:00:00 | t

Success!, Marge’s account has now been marked expired, since she has not logged on in the last 30 days.

Your first inclination may be to extract the day from age, but this does not work as one may think. Take for example the following…
testdb=# select age(now(), CURRENT_DATE-31); age ----------------------------- 1 mon 1 day 13:02:30.065623 (1 row)

Now if we extract day, this is what we get…
testdb=# select extract(day from age(now(), CURRENT_DATE-31)); date_part ----------- 1 (1 row)

This is not in fact what we were looking for, if we were to use this method to expire accounts, this account appears to have been logged in as recently as one day ago, when in fact 31 days have elapsed. So this is why we chose to use epoch, which in this case will get total elapsed seconds.

Now, let’s review how you might write this for MySQL.
DELIMITER // CREATE PROCEDURE mark_expired (days INT) BEGIN UPDATE users SET expired = true WHERE last_login < SUBDATE(CURRENT_TIMESTAMP, INTERVAL days DAY); END // DELIMITER ;
In this example we take a slightly different approach, we subtract the days argument from the current timestamp, and see if the last login timestamp is older than this. This leverages MySQL’s SUBDATE function and avoids having to convert days into seconds.

Let’s take a look at our table again, before calling the procedure
mysql> select * from users; +----------+---------------------+---------+ | username | last_login | expired | +----------+---------------------+---------+ | homer | 2008-12-18 00:00:00 | 0 | | marge | 2008-11-18 00:00:00 | 0 | | bart | 2008-12-14 00:00:00 | 0 | | lisa | 2008-12-17 00:00:00 | 0 | | maggie | 2008-12-19 00:00:00 | 0 | +----------+---------------------+---------+

And how you would call the procedure in MySQL.
mysql> CALL mark_expired(30); mysql> select * from users; +----------+---------------------+---------+ | username | last_login | expired | +----------+---------------------+---------+ | homer | 2008-12-18 00:00:00 | 0 | | marge | 2008-11-18 00:00:00 | 1 | | bart | 2008-12-14 00:00:00 | 0 | | lisa | 2008-12-17 00:00:00 | 0 | | maggie | 2008-12-19 00:00:00 | 0 | +----------+---------------------+---------+

Again, success, Marge’s account has been expired.

Importing XML into a database with Python and SQLAlchemy

tgc — Tue, 29 Jan 2008 21:19:47 +0000

Let’s begin by analyzing the XML we want to import into our database, it consists of a book’s ISBN, title, and author.
The Consumer M. Gira The Wind-Up Bird Chronicle Haruki Murakami

Now we can create the database table.
create table books ( isbn varchar(14) primary key not null, title varchar(50), author varchar(50) );

Our example depends on SQLAlchemy, which is a “Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.”

While our example leverages a Postgres database, given SQLAlchemy’s database agnostic approach, it can trivially be modified to access a MySQL database.

This code is derived from a SAX parser originally found in Programming Python 3rd Edition.

# bookhandler.py from sqlalchemy import * from sqlalchemy.orm import * import xml.sax.handler pg_db = create_engine('postgres:///testdb?user=homer') metadata = MetaData(pg_db) books_table = Table('books', metadata, autoload=True) class Book(object): pass mapper(Book, books_table) class BookHandler(xml.sax.handler.ContentHandler): def __init__(self): self.buffer = "" self.inField = 0 self.session = create_session(bind=pg_db) def startElement(self, name, attributes): if name == "book": self.isbn = attributes["isbn"] elif name == "title": self.inField = 1 elif name == "author": self.inField = 1 def characters(self, data): if self.inField: self.buffer += data def endElement(self, name): if name == "book": self.session.begin() self.newbook = Book() self.newbook.isbn = self.isbn self.newbook.title = self.title self.newbook.author = self.author self.session.save(self.newbook) self.session.commit() elif name == "title": self.inField = 0 self.title = self.buffer elif name == "author": self.inField = 0 self.author = self.buffer self.buffer = ""
Now that we’ve set up our sax parser and handler to parse and load the entries from books.xml into the table, lets set up a small script to drive it:
# runit.py import bookhandler import xml.sax parser = xml.sax.make_parser() handler = bookhandler.BookHandler() parser.setContentHandler(handler) parser.parse("books.xml")

Now let’s see if it works:
$ ls bookhandler.py books.xml runit.py $ python ./runit.py $ psql testdb testdb=# select * from books; isbn | title | author ---------------+----------------------------+----------------- 1-880985-26-8 | The Consumer | M. Gira 0-679775-43-9 | The Wind-Up Bird Chronicle | Haruki Murakami (2 rows)

Importing data from a file into a PostgreSQL database

tgc — Mon, 06 Aug 2007 14:21:13 +0000

To follow up on our previous article on importing data into a MySQL database, is how to Import data from a file into a PostgreSQL database.

In this example, we’ll focus on the data contained here, in an IP-to-Country csv file. This data consists of the first IP address in the range (in long format), the last IP address in the range (again, in long format), and the alpha-2, alpha-3 and country name (according to ISO 3166-1) of the country which the IP range is assigned to.

Here is a brief excerpt to get an idea of the information we’ll be importing.
"3739572224","3739574271","AU","AUS","AUSTRALIA" "3739574272","3739680767","JP","JPN","JAPAN" "3739680768","3739697151","KR","KOR","REPUBLIC OF KOREA" "3739697152","3739746303","JP","JPN","JAPAN" "3739746304","3740270591","KR","KOR","REPUBLIC OF KOREA" "3740270592","3740925951","CN","CHN","CHINA" "3740925952","3741024255","TW","TWN","TAIWAN" "3741024256","3741057023","KR","KOR","REPUBLIC OF KOREA" "3741057024","3741319167","VN","VNM","VIET NAM" "3758096384","4294967295","US","USA","UNITED STATES"

Let’s start by creating the table we need to support this data.
ipdb=# create table ip_to_country ( startrange bigint, endrange bigint, country_alpha2 varchar(2), country_alpha3 varchar(3), country varchar(50) ); CREATE TABLE

In the csv above, you’ll notice that the fields are comma separated and also enclosed in double quotations. Postgres’ COPY function allows you to define what the DELIMITER and QUOTE are, among other variables, you can type \h copy at the psql prompt to see all the arguments COPY takes.

Here is how we import our ip-to-country data.
ipdb=# copy ip_to_country from '/tmp/ip-to-country.csv' WITH DELIMITER AS ',' CSV QUOTE AS '"'; COPY 79440

There were 79440 entries copied, a quick line count on the file can help you determine if all the data was imported.

Now let’s see if it imported correctly, we’ll take the last entry in the above excerpt and query for the startrange
ipdb=# select * from ip_to_country where startrange = 3758096384; startrange | endrange | country_alpha2 | country_alpha3 | country ------------+------------+----------------+----------------+--------------- 3758096384 | 4294967295 | US | USA | UNITED STATES (1 row)
Perfect, and as you’ll notice the quotes enclosing the data were stripped during the import as well.