Postgresql Server Side Cursor

When a database query is executed, the Psycopg cursor usually fetches all the records returned by the backend, transferring them to the client process. If the query returned an huge amount of data, a proportionally large amount of memory will be allocated by the client.

If the dataset is too large to be practically handled on the client side, it is possible to create a server side cursor. Using this kind of cursor it is possible to transfer to the client only a controlled amount of data, so that a large dataset can be examined without keeping it entirely in memory.

Server side cursor are created in PostgreSQL using the DECLARE command and subsequently handled using MOVE, FETCH and CLOSE commands. postgresql-cursor

Cursor

Psycopg wraps the database server side cursor in named cursors. A named cursor is created using the cursor() method specifying the name parameter.

1. using DECLARE command create named cursor (note: declare must be in transaction)
isnp=# declare xxxx CURSOR WITHOUT HOLD FOR select * from citys;
ERROR:  DECLARE CURSOR can only be used in transaction blocks
isnp=# fetch xxxx;
ERROR:  cursor "xxxx" does not exist
isnp=# begin;
BEGIN
isnp=# declare xxxx CURSOR WITHOUT HOLD FOR select * from citys;
DECLARE CURSOR
isnp=# fetch xxxx;
    created_date        |        updated_date        |   id   | level |  name  | parent_id 
----------------------------+----------------------------+--------+-------+--------+-----------
 2016-09-09 15:10:47.291513 | 2016-09-09 15:10:47.291513 | 410000 |     1 | 河南省 |          
(1 row)

isnp=# fetch xxxx;
    created_date        |        updated_date        |   id   | level |  name  | parent_id 
----------------------------+----------------------------+--------+-------+--------+-----------
2016-09-12 15:10:29.192463 | 2016-09-12 15:10:29.192463 | 410100 |     2 | 郑州市 |    410000

2. using function return cursor
isnp=# create function myfunction(refcursor) returns refcursor as $$
isnp$# begin
isnp$# open $1 for select * from citys;
isnp$# return $1;
isnp$# end;
isnp$# $$
isnp-# language plpgsql;
CREATE FUNCTION
isnp=# begin;
BEGIN
isnp=# select myfunction('mycursor');
myfunction 
------------
mycursor
(1 row)

isnp=# fetch mycursor;
    created_date        |        updated_date        |   id   | level |  name  | parent_id 
----------------------------+----------------------------+--------+-------+--------+-----------
2016-09-09 15:10:47.291513 | 2016-09-09 15:10:47.291513 | 410000 |     1 | 河南省 |          
(1 row)

isnp=# fetch mycursor;
    created_date        |        updated_date        |   id   | level |  name  | parent_id 
----------------------------+----------------------------+--------+-------+--------+-----------
2016-09-12 15:10:29.192463 | 2016-09-12 15:10:29.192463 | 410100 |     2 | 郑州市 |    410000
(1 row)

isnp=# fetch mycursor;
    created_date        |        updated_date        |   id   | level | name | parent_id 
----------------------------+----------------------------+--------+-------+------+-----------
2016-09-12 15:10:29.194794 | 2016-09-12 15:10:29.194794 | 410101 |     3 | 直属 |    410100
(1 row)

Python Code

1. psycopg2 example
import psycopg2
# server side cursor via function method
connection = psycopg2.connect("dbname=isnp")
cursor = connection.cursor()
cursor.callproc("myfunction", ["xxxx"])
cursor1 = connection.cursor("xxxx")
print(cursor1.fetchmany(100))
cursor1.close()
connection.close()

2. sqlalchemy example
from sqlalchemy import engine_from_config

config = {
    "sqlalchemy.url": "postgresql:///isnp",
    "sqlalchemy.echo": True,
    "sqlalchemy.server_side_cursors": True,
}
engine = engine_from_config(config)

connection = engine.connect()
proxy_results = connection.execution_options(stream_results=True).execute("select * from citys")
print(proxy_results.fetchmany(10))