Friday, October 31, 2008

This close to a datastore --> json server on google apps engine

Phew ... a good afternoon's hacking.

I am *this close* to completing a very basic prototype of a json server for google apps engine. The idea is that you visit a URL which contains the query you want to evaluate (hence my previous post) and get back a JSON encoding of the data. Obviously, that's for requests only.

However, what it *means* is that you can then write an AJAX client which can interface directly to a GAE data store. And *that* is a good thing.

Sure, it's about as functional as a car with no steering wheel, but in principle it works. I have achieved the following goals:
* Store some data in the data store
* Retrieve it with a GQL query
* Write a web server which decodes a query from the URL in hex
* Use that to retrieve said data from the query store
* Render the data to a web page in HTML
* Render the a json string to the web page (or json-looking, anyway)

The following goals remain:
* Write a version which emits the json string with appropriate mime type
* Write a javascript client which retrieves the data and renders it in a web page
* Put up some vaguely comprehensible web pages explaining the thing and demonstrating its use
* Make the store/retrieval more general by:
-- Not relying on a specific database structure
-- Going beyond string-only data
-- Do something about INSERT queries

I'm having some fun with this. Anyone who is interested, let me know and I can keep you posted of progress. I think it should be really useful.

Efficient encoding of URL arguments

I would like to have URLs which look like:
"" or something similar. That last part should be some reasonably space-efficient encoding of something like "SELECT * from FOO where CONDITION"

I have found I can get part-way using Python's encode and decode functions, e.g.
encoded = "SELECT * from FOO where CONDITION".encode("hex")
decoded = encoded.decode("hex")

However, the resulting values are still a bit long. This method quite nearly removes nonprintable characters, spaces etc from the URL. However, I have this nagging feeling that I could be doing some compression along the way to make the encoded part shorter. For example, HEX uses only part of the alphanumeric space. I'd like to use the digits 0-9 and letters a-z and A-Z ideally.

Let's just make this clear -- for no very good reason that I can justify, I want to encode a query as the shortest possible string using printable, alphanumeric characters (and no spaces) and be able to simply decode the string also.

"The Internet" has told me about md5 hashing, but I need a two-way function here. The help on string.encode in Python simply says I can specify the name of any encoder, but I can't find a list of the default encoders available.

Any ideas, internet?

Cheers and thanks in advance,

Thursday, October 30, 2008

OSDC 2008 pre-conference activities

I notice that the OSDC conference days have dropped from the 1st - 5th
to the 2nd to the 5th as the conference schedule has been firming up.
I find myself now with a free day on the 1st. I wonder if there are
any Pythonistas in Sydney who would care to catch up on that day for a
discussion group, BoF sessions or anything else? Looking at the
program, it seems to have fewer of such events than past conferences.
There are some available slots for these sessions, but if anyone wants
to get together for some more in-depth discussions, I'd be up for it.

I'd be very interesting in informal sessions on the following topics:
* Open source in government
* Weather forecasting
* AJAX development
* Artificial intelligence
* Advanced Python skills
* Open Source & Project Management

I have also posted this message to various Australian Python mailing lists. So far there are two takers for some activities on the Monday. If interested parties can email me, I will see if there are enough takers to organise some small-group discussions on topics of interest. So far the takers are 1 for Open Source in Government and another for AI.

-Tennessee Leeuwenburg

Wednesday, October 29, 2008

Automatic text generation is live!

Hi all,

The system which I have been working on for three years has just gone live. My component of this system generates text forecasts automatically, based on gridded forecast data which is initialised from model conditions and then manipulated by weather forecasters. Any particular text forecast may also be edited by the forecasters after generation, but the bulk of the forecasts are published without forecaster intervention in the text.

Examples include the Melbourne forecast,

A number of other Victorian town forecasts, Victorian district forecasts and an expanded set of Precis and Town forecasts are also produced. Also, a larger number of forecast days are included for many of these locations.

This page ( describes some of the changes to service which come with the new forecast system.