Howto run web2py apps on GAE (Google App Engine)

requires version in trunk

As you know web2py runs on almost any architecture and supports transparently many DB back-ends (SQLite, MySQL, PostgreSQL, Oracle).

Much of web2py runs on GAE too. Here we will try to clarify what runs, what does not run, and what to do about it.

The main issue with GAE is the absence of file system access. This means you cannot store session, tickets and uploaded files on the filesystem. They have to go in the databases.

GAE provides its own database based on BigTables. The Web2py ORM can access the GAE database and automatically generates GQL (Google Query Language) queries, but GQL has limitations compared with SQL and reflect in limitations in the types of ORM queries one can build.

Example

Consider the following application that DOES NOT RUN on GAE:

model db.py:

db=SQLDB("sqlite://db.db")

db.define_table('image',
  SQLField('title'),
  SQLField('filename','upload'))

db.image.title.requires=IS_NOT_EMPTY()

controller default.py

def index():
    form=SQLFORM(db.image)
    if form.accepts(request.vars): response.flash='record inserted'
    images=db().select(db.image.ALL,orderby=db.image.title)
    return dict(form=form,images=images)

def edit():
    image=db(db.image.id==request.args[0]).select()[0]
    form=SQLFORM(db.image,image,upload=URL(r=request,f='download'))
    if form.accepts(request.vars): redirect(URL(r=request,f='index'))
    return dict(form=form)

def download():
    import os
    from gluon.contenttype import contenttype
    image=db(db.image.filename==request.args[0]).select()[0]
    response.headers['Content-Type']=contenttype(image.filename)
    return response.stream(os.path.join(request.folder,'uploads',image.filename)

view default/index.html

{{extend 'layout.html'}}
<h2>Post image</h2>
{{=form}}

<h2>Images</h2>
<ul>
{{for image in images:}}
   <li>{{=image.title}}
   [<a href="{{=URL(r=request,f='edit',args=[image.id])}}">edit</a>]
   <br/>
   <img width="200px" src="{{=URL(r=request,f='download',args=[image.filename])}}"/>
   </li>
{{pass}}
</ul>

view default/edit.html

{{extend 'layout.html'}}
<h2>Edit image</h2>
{{=form}}

It allows visitors to upload images with a title and to [edit] those title, images. All images are shown in the index page.

This app does not run because:

uses SQLDB
uses file based sessions
uploads files in the filesystem

Porting to GAE

In order to port it to GAE you need to change

db=SQLDB("sqlite://db.db")

into

try:
    from gluon.contrib.gql import *
    db=GQLDB()
except:
    db=SQLDB("sqlite://db.db")
session.connect(request,response,db=db)

The last line tells web2py to store sessions in DB. A web2py_session table is created automatically.

You need to change the model to

db.define_table('image',
   SQLField('title'),
   SQLField('filename','upload',uploadfield='data'),
   SQLField('data','blob',default=''))

i.e. for every field of type upload, add a new blob field where to upload the file into. The blob field must default to an empty string.

You also need to replace the last line of the download controller function from

return response.stream(os.path.join(request.folder,'uploads',image.filename)

return image[db.image.filename.uploadfield]

So that you retrieve files from the db instead of the filesystem.

That's it! Code written for GAE also works without GAE since it is valid web2py code.

On Complex Queries

These queries will work on GAE

db(db.image.id==1).select()
db(db.image.id==1).delete()
db(db.image.id==1).update(title='title')

db(db.image.title<'z').select()
db(db.image.title>'a').delete()
db(db.image.title=='test').update(title='title')

db((db.image.title<'z')&(db.image.title>'a')).select(db.image.ALL,sortby=db.image.title,limitby=(0,10))

but on GAE you WILL NOT BE ABLE to:

use OR (operator "|") in queries
use the not operators ("!=" and "~")
use LIKE (operator "like") and IN (operator "belongs")
use groupby
mix queries by id and queries by other fields like in (db.image.id==id)&(db.image.title=='')
manipulate date, time, datetime
select anything other than db.table.ALL
do JOIN; i.e. build queries involving more than one table

What else does not work

The following functionalities do not work:

admin
tickets (they are logged but not stored in files)

We are working to resolve some of these issues. Admin is available when doing local development.

Comments

Dale on 2008-06-19 14:54:11 says
This looks very interesting. What is your take on the GAE? [inappropriate?]

web2py AlterEgo