My Own Hat

Why doesn't my circuit work?

2012-01-31T12:31:00.000+11:00

Okay, so I finally managed to build something that didn't work! This is actually neat, since it represents a problem-solving challenge to be overcome, and something that goes beyond colouring in the lines. Well, apparently I couldn't colour in the lines, which is how I got here.

It looks like a fairly simple circuit. Basically, the motor is connected to ground at one end, and via a transistor to a 5V source at the other. Pin 9 there goes to the transistor, so pulsing that slowly ought to turn the motor on and off. There is also a diode, and I don't know what the hell it's for. To be more accurate, I don't know why it's necessary here, or what it's doing in this circuit. In general, the concept seems straitforward enough. I'm also not 100% confident I have the diode, the transistor or the various resistors the right way around.

The resistors, I'm pretty sure, could get plugged in backwards and still operate normally. The diode I realise will simply not allow current passed if plugged in backwards. Given that if I skip the transistor the motor spins, the diode 'must' be the right way in (assuming it's working correctly). So the fault seems to be with the transistor.

However, no matter how I connect the transistor, it doesn't work correctly. One way, the motor is dead. The other way, the motor spins constantly. Total ignoring of the collector (Pin 9 there). Question: is it possible to break a transistor by sending current the wrong way through? How about with diodes?

Now, I did check. The transistor in NPN, which as far as I can tell translates as "on by default", so there is one explanation: Pin 9 is broken. I tried Pin 6, no luck. So I tried coding up a flashing LED to match the Pin 9 signal. The LED flashes as required, so unless both Pin 9 and Pin 6 are physically damaged on the Arduino, it shouldn't be that.

So, now I'm stuck. Without a multimeter (or maybe just some LEDs and alligator clips) I'm pretty much unable to determine which parts of my circuit are receiving the expected current. Did I short something out? Is my diode in backwards? What current is coming out of Pin 9?

Answers to all these exciting newbie questions, and more, in weeks to come.

In the meantime, to all you electrical engineers out there, I hope you enjoyed this brief sojourn into electrical bafflement :)

PS: Some people who spotted these photos on G+ have pointed out that the diode is likely present to protect against a back-spike of power coming FROM the motor as it spins down.

Assembling the chassis

2012-01-28T21:42:00.001+11:00

This post is going to document the assembling of the chassis. I purchased the Magician Chassis as the basis for my bot. Assembly was mostly straightforward, although I did need to do just a little hackwork along the way to bring it together. The finished product looks like this:

It's not the greatest photo ever, but you get the idea. There are two motors, one attached to each large wheel. There is an additional rear "trackball" thing which bears the weight of the rear end. The construction is sturdy-ish. Certainly it doesn't seem like it's going to break during construction, but it should definitely not suffer a drop.

The kit comes with instructions, which kind of mostly cover what you need to know. In addition to the instructions and this blog post, there is a similar blog post available here: http://www.hobbytronics.co.uk/magician-chassis-build. It's probably more useful than this post, but this is my story :)

Here is a photo of the kit parts, unassembled:

The process begins by attaching the motors. This turns out to be medium level tricky, where easy is "goes according to the instructions" and hard would be "I basically had to MacGuyver it from first principles". My main issue was there there are some little red things which look like gears which attach to a spindle on the side of the motor. THESE APPEAR TO HAVE NO FUNCTION. Even after assembly, no function.

You can see the red thing here. One of my red things needed to have the whole expanded carefully with a Stanley knife in order to accept the spindle. Bolting the struts on was not too bad, although it's a darn tight fit. I had to rotate the struts 180 degrees even though they look pretty symmetrical. They do go on. Anyone who knows what those red things are for, please let me know.

Attaching the wheels is a matter of shoving them onto the outside spindle of each engine. They don't fit very well (no satisfying push... click). So I hope they don't come off, but it seems okay.

Attaching the trackball is super easy. No worries there, although I might lubricate the socket at some point. The base is now completed! The leads running off the motors strike me as a bit of a catching hazard, I think it would be better to put some electrical tape on to help run them up to the upper level where they will attach to the Arduino later on, but I'll do that at another point in time.

The next step is to put the base right-way-up and attach the battery housing. The battery housing is terrible quality, and in a stupid place. The worst thing about it is that the screw-holes are so close to the edge that you can't actually put a screw in. I had to cut down the edges with a Stanley knife. I hope this isn't a problem later.

See the screw? You can see it through the hole I cut to allow the head through.

From here, it's a doddle to finish assembly. Screwing on the spacers and the top level is super-easy, leading to the finished product:

As you can see, the battery housing is in a near-inaccessible location. The HobbyTronics post I linked to earlier said they pulled it out and housed it on the top level, which certainly makes sense. However, I figure I should follow the rules before breaking them, so there is the finished product!

Of course, right now it doesn't DO anything, because I haven't wired up any power or attached the Arduino to control the motors. However, it was a great evening's work and took a little under two hours from fetching my equipment to a finished product and a clean workbench again.

Some inspirational material

2012-01-20T17:49:00.000+11:00

This video does a fantastic job of expressing the sentiment and excitement around building a basic autonomous robot....

http://t.co/6ioJYoCI

Thanks, that guy! (http://www.instructables.com/member/Brandon121233/)

It's here! It's here!

2012-01-16T12:57:00.000+11:00

Okay, so now my Arduino journey can begin in earnest. Why now? Because it arrived today! Squeee!

So, I thought a short unboxing post was in order. Obviously, I couldn't possibly wait until this evening at home to open it, so here are my on-the-desk photos. I'll follow up with some more later when I've actually had the opportunity to perform some construction or investigation.

Inside the SparkFun Inventor's Kit

The great, the only, Arduino Uno

I haz a "Magician Chassis", a "Sparkfun Inventor's Kit" and a "Soldering Iron". I presume that's enough to get in trouble with....

Amateur robotics: getting started

2012-01-12T02:45:00.000+11:00

So, here's the start of a blog series which I plan to use to document my tinkering with my soon-to-arrive Arduino board. For those who don't know, Arduino is an open-source electronics, um, computer board thingy with lots of i/o pins you can used for sensors and effectors. It can also be hooked up to a mobile phone (or cellphone for any North American readers), which can interface with the Arduino board. This can be done using Java, or even Python (thanks to the SL4A interpreter). No phone hacking required!

So I intend to thoroughly document my process, and share it here. This is really for myself, but anyone else who is interested in amateur robotics might be interested in following along my story as a complete newbie at this stuff.

So, the beginning...

I just placed my order for the following items:
-- http://littlebirdelectronics.com/products/sparkfun-inventors-kit-for-arduino
-- http://littlebirdelectronics.com/products/magician-chassis
-- http://littlebirdelectronics.com/products/ioio-for-android
-- http://littlebirdelectronics.com/products/infrared-proximity-sensor-short-range-sharp-gp2d120xj00f

My hope is to be able to build something like this:
-- http://www.youtube.com/watch?feature=player_detailpage&v=9cVSzB8otpU#t=13s

But I'll have to start right from the very beginning:
-- http://www.youtube.com/watch?v=fCxzA9_kg6s

Self-deprecating code

2012-01-11T16:02:00.002+11:00

I want code that will raise a warning if a method is called that hasn't been used in six months. That would be awesome. By this I mean:
-- A log of unused methods
-- A log of uses of crufty old methods which might not be reliable

An idea: an anti-framework for science

2011-11-29T14:44:00.000+11:00

So here's my idea: an anti-framework. I'm interested in the idea of building a framework, because it would be nice to provide a nice, standard set of libraries and modules to work in (much like the standard library, but extended to cover the scientific domain). However, the problem with building frameworks is basically the competing standards problem.

So here's my new idea. An anti-framework, build out of a relatively small number of de facto standard scientific components for Python (like numpy, scipy, matplotlib, I imagine there are some others) and call it something zingy. Any framework or environment which passes its unit tests in the anti-framework is then "zingy-name-compatible". You don't run with a zingy-name framework, because lots of people need more than what's in the standard framework for all sorts of reasons. It's perfectly okay for some-other-framework to extend zingy-name-framework, so long as it remains backwards-compatible with zingy-name framework.

Then, anyone building a relevant application, framework, or whatever can be zingy-compatible if they will support any tool written only using the zingy core libraries.

"Inspiration is for amateurs; the rest of us just show up and get to work."

2011-09-07T20:17:00.000+10:00

I came across this fantastic quote today, capturing something which I've always found to be true. I think I can now add this as a second item to the list of things I find to be true in the workplace. The first is that the relentless pursuit of simplicity is the only weapon we have against the complexity of what we are trying to achieve.

“The advice I like to give young artists, or really anybody who’ll listen to me, is not to wait around for inspiration. Inspiration is for amateurs; the rest of us just show up and get to work. If you wait around for the clouds to part and a bolt of lightning to strike you in the brain, you are not going to make an awful lot of work. All the best ideas come out of the process; they come out of the work itself. Things occur to you. If you’re sitting around trying to dream up a great art idea, you can sit there a long time before anything happens. But if you just get to work, something will occur to you and something else will occur to you and somthing else that you reject will push you in another direction. Inspiration is absolutely unnecessary and somehow deceptive. You feel like you need this great idea before you can get down to work, and I find that’s almost never the case.” ~ Chuck Close

Sourced from: http://blog.jacquihocking.com/2011/09/07/since-cambodia-the-beginning-of-a-new-journey/

Debug my tutorials (and try something new)

2011-07-26T15:38:00.000+10:00

https://bitbucket.org/tleeuwenburg/benchmarker.py/wiki/Home

As I've mentioned before, I'm working on a new project for helping collect benchmarks for Python programs. To start with, only CPU performance is measured since that's the easiest data to collect, but I'd like to collect other metrics eventually. The wiki has three fairly thorough tutorials, and some more under construction. The installation of the package with easy_install seems to be working okay, and I believe all the tutorials have correctly functioning code snippets.

If you have an interest in collecting performance data on your Python program, maybe you would be happy to work through the tutorials and let me know if you come across any bugs in benchmarker.py, or in the tutorials. If you have any questions that you would like to see addressed in the tutorials, please also let me know!

If you want to cut straight to the action, you can install benchmarker.py with

easy_install benchmarker.py

(although I'd suggest doing this in a virtualenv for the time being). Please let me know if that doesn't work -- this whole exercise will be somewhat undercut if others have trouble installing the package!

Thanks very much to those who have already helped identify bugs and issues, and contribute to lifting the standard of this package.

-Tennessee

Help testing my new package-in-progress

2011-05-08T15:18:00.000+10:00

Could some helpful Pythonista please try:

easy_install benchmarker.py

and let me know if it installs for you? If it installs, here's a simple script to use it and see if it works... this script is a pretty thin usage of the package but proves it works in a basic sense:

>>> import bench
>>> from bench.benchmarker import benchmark
>>> @benchmark()
... def time_me():
... for _ in range(100):
... pass
...
>>> time_me()
>>> bench.benchmarker.print_stats()
Sat May 7 22:06:06 2011 /tmp/tmp.pstats

   300 function calls in 0.002 CPU seconds

   Random listing order was used

   ncalls tottime percall cumtime percall filename:lineno(function)
   100 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
   100 0.001 0.000 0.002 0.000 :1(foo)
   100 0.001 0.000 0.001 0.000 {range}

Thanks!
-Tennessee

python-twitter-tools wins!

2011-01-23T22:36:00.000+11:00

(1) install python-twitter-tools
(2) Register an app with twitter to get oath credentials
(3) Run this code ftw!
(4) You'll need to set those four variables to the ones you got in step 2 for this to work of course

python-twitter-tools has the most natural twitter API I've seen. It's really beautiful. The namespace follows the restful API urls, so you don't need any documentation other than what's on the twitter dev website. It really does Just Work.

import twitter

if __name__ == "__main__":

   oa = twitter.oauth.OAuth(access_key, access_secret, consumer_key, consumer_secret)
   tw = twitter.Twitter(auth=oa)

   followers = tw.statuses.followers()

   print "You have", len(followers), "followers"

   for f in followers:
   print f['name']

Three concentric spherical surfaces, with a randomly-connected graph overlaid on each one, spinning.

2011-01-22T21:38:00.000+11:00

Three concentric spherical surfaces, with a randomly-connected graph overlaid on each one, spinning. http://www.youtube.com/watch?v=bXOTCZXkUjk ... born of an idea I wanted to explore. Python + Mayavi = rapid gratification! Thought I'd post the code for anyone who's interested. I'm trialling the use of the Enthought Python Distribution, it certainly provides a nice technology stack to work with. All the components could be installed separately, but I'm very happy to have something where things "just work" (at least, so far).

from enthought.mayavi import mlab
from random import random
from math import sin
from math import cos
import math

radius = 1
azimuth = 0
inclination = 0

x, y = 0, 0

RAD1 = 1
RAD2 = 2
RAD3 = 3

def coord_from_rai((r, a, i)):

   x = round(r * sin(i) * cos(a), 4)
   y = round(r * sin(i) * sin(a), 4)
   z = round(r * cos(i), 4)

   return ((x,y,z))

def rai_from_coord((x, y, z)):

   rad = math.sqrt((x**2 + y**2 + z**2))
   azimuth = math.atan(y/max(x,0.00000000000000001))
   inclination = math.acos(z/rad)

   return((rad, azimuth, inclination))

def test_point_spherical_conversions():

   points = [(1.,0.,0.),
   (0., 1., 0.),
   (0., 0., 1.),
   (1., 1., 1.),
   (0.3,0.5,0.9),
   (2.6,2.2,2.3),
   (2.,3.,4.),
   (5.,7.,8.),
   ]
   for point in points:
   try:
   assert coord_from_rai(rai_from_coord(point)) == point

   except:
   print "Original Point: ", point
   print "Rad, Az, Inc: ", rai_from_coord(point)
   print "Converted Coord: ", coord_from_rai(rai_from_coord(point))
   raise

def frange(start, stop, inc):

   stop = stop * 100

   for i in range(start, stop, inc * 100):
   return float(i) / 100.

def points_for_line_arc(point1, point2, coords):

   rad1, azimuth1, inclination1 = point1
   rad2, azimuth2, inclination2 = point2

   delta_rad = (rad2 - rad1) / 15.
   delta_az = (azimuth2 - azimuth1) / 15.
   delta_inc = (inclination2 - inclination1) / 15.

   line = []
   line.append(point1)

   for i in range(1, 15):
   (rad, az, inc) = line[-1]
   rai = (rad + delta_rad, az + delta_az, inc + delta_inc)
   line.append(rai)

   if line[-1] != point2:
   line.append(point2)

   return line

def get_unique_random_point(radius, coords):
   azimuth = 2 * math.pi * random()
   inclination = 2 * math.pi * random()

   return ((radius, azimuth, inclination))

def plot_randomly_connected(radius, count, color=(1., 1., 1.)):
   '''
   Plot points on a sphere distance from the origin

   Points will be connected by an arc line with 70% probability, with 5
   subpoints on the arc line
   '''

   coords = []
   for _ in range(0, count):
   coord = get_unique_random_point(radius, coords)
   coords.append(coord)

   if len(coords) > 1 and random() > .001:
   line = coords[-2:]
   line = points_for_line_arc(line[0], line[1], coords)
   cartesianLine = [coord_from_rai(point) for point in line]
   (x, y, z) = zip(*cartesianLine)
   mlab.plot3d(x, y, z, opacity=0.05, tube_radius=0.025, color=color)

   cartesianPoints = [coord_from_rai(point) for point in coords]
   (x, y, z) = zip(*cartesianPoints)
   mlab.points3d(x, y, z, color=color, opacity=0.8/radius, scale_factor =1./(2*radius))

plot_randomly_connected(1, 4, color=(.7,.2,.2))
plot_randomly_connected(2, 16, color=(.2,.7,.2))
plot_randomly_connected(3, 32, color=(.2,.2,.7))

#test_point_spherical_conversions()

#mlab.show()
from enthought.mayavi import mlab
@mlab.animate
def anim():
   f = mlab.gcf()
   while 1:
   f.scene.camera.azimuth(4)
   f.scene.render()
   yield

a = anim() # Starts the animation.
mlab.show()

Service-based URL shorteners

2010-12-10T16:14:00.000+11:00

Generic URL-shortening services like goo.gl and bit.ly, should be made service-specific (image.ly, tube.ly) to avoid malware site redirects.

It would work like this. Hypothetical image-based URL shortener image-bit.ly would check to make sure the site pointed to met any of the following qualifiers:
-- Content of URL is a recognised image format. Require proper mime-type and image headers.
-- Content of URL is a recognised image service provider, e.g. Facebook photo, Flickr photo etc

This would prevent people doing something like the following:
"Hey check out this awesome photo of a rabbit SCUBA diving http://bit.ly/roLeb"

which actually redirects you to something else entirely, like a shopping website, or worse something seedy, a malware site, or actual criminal content.

A similar hypothetical tube-based URL shortener, pg-tube-bit.ly would do the same thing, but check the content was hosted on YouTube, or a variety of other family-friendly video hosting sites.

That way, you still get URL shortened goodness, but also content safety.

That's all!
-Tennessee

Idea: Crowdcasting News

2010-12-07T09:06:00.000+11:00

Okay, here's how I want to my get my news now. Think of it as like Digg meets the social graph.

I have some kind of trust network of friends. A vanilla graph would do, but let's face it, I value news from some sources more than other. Anyone who stumbles on some news (or generates some) might tweet it at the moment, or Facebook it. I want them to amplify it instead.

There is a database somewhere that tracks all the amplification clicks it gets against URLs, say keeping track of 2-3 days information. It then extracts those clicks which originated from my trust network, then weights all the URLs according to number of clicks. If a URL meets threshold, I get it as news. In fact, what I specify is my desired number of news items per day, and the threshold adapts.

If I had that, I would completely replace newspapers. I would have an aggregated news feed and an aggregated comment feed (currently newspapers+slashdot+dig, and facebook+twitter+email).

I think that would be cool.

Daft ideas on social graphs

2010-11-19T12:59:00.000+11:00

Here's my issue with Facebook security. It's not information bleed to third parties, which I agree is important, but information bleed amongst social networks that I would like to keep separate. Work/Social being the primary division I would like to maintain. It's a lot *less* important to me if some marketing firm has my information than if my workmates know I faked sick leave or my friends are showered with geek updates. I would like multiple social graphs please. And I would like to give my blessing on a post or other Facebook object to a graph at a time, not a person at a time or an application at a time. I'm happy with connecting with applications and groups using "Like" or "Join", and I'm happy with friends-of-friends seeing most of my photos/updates. Within a single graph.

I would like the following graphs:

The supergraph. Everyone in all of my graphs.
A professional subgraph. Workplace + geek.
A social subgraph: Friends and family.
  - A family sub-subpgraph
  - Logically, I may also need a friends sub-subgraph
  - A dark subgraph: My connections are invisible to others in the graph

Maybe I also want a public broadcast faux graph.

That way, for example, my "Farmville Lame App" could suck down my social subgraph permissive fields, but be denied information on how much I earn. Why? Because it only gets fed what I feed my social graph.

+1 for xmind... download and use it, it's awesome!

2010-07-29T10:38:00.000+10:00

http://www.xmind.net/

I've just downloaded this, and I've used it to create about 9 diagrams in two days. It's totally awesome. It can be used to create the following charts with no fuss at all, with nice icons if you would like to include that sort of thing:
  -- Brainstorming map
  -- Organisation chart
  -- Spreadsheet-style chart
  -- Fishbone chart

The concept map style chart really helped me to capture the major tasks, goals, responsibilities and relationships for me and my team, bringing together kinds of information that are otherwise quite difficult to relate to one another. It really helped me talk to them about how things tie together.

I've also used the other chart styles to map out processes, systems and challenges, showing my boss and various colleagues how things hang together.

Review of the Nexus One

2010-07-13T13:15:00.000+10:00

Okay, so the Nexus One has just been released in Australia, and I am now a proud owner. This is my first foray into Phones 2.0, so I wasn't quite sure what to expect. I also have insanely high standards, and the quality of the computing platform comes into my evaluation.

Review: Three and a Half Stars / Pretty good / Somewhat exceeds expectations

Based on the iPhone4 articles I've read, antennae issues aside, the iPhone4 would rank at 4 stars. (out of 5 in both cases).

The Nexus One has a good enough processor. It successfully gets over the bar of being an acceptable computing environment. I believe it's a 1Ghz chip, and I would say this is about the minimum you'd want before regarding a phone as a proper computing platform.

The best thing about the Android is that it is open source. This means you can install stuff yourself, use the phone to do what you want, access any websites you and and basically freely use your own property. I've already used this to my own advantage and this was an overriding concern for me. That's where my perception of the iPhone lost a star.

There are a few negatives I can report. The battery life is lousy. I'd expect this to be true of all the competitors, but it's just annoying to have to recharge twice a day (if you are using it somewhat actively). I don't think it's Google/HTC's fault as such, but it's an annoyance which I expect is present on every new gen phone.

It doesn't have a front-facing camera, which means no video calls and no potential for cool face-recognition software. It could do with an extra camera.

It doesn't do such a great job of flash. It will play YouTube (thank goodness) but if you stray to other sites with flash payloads, you'll be in trouble. I know "it's coming" but it's not there yet.

The side buttons are a bit annoying. I accidentally push one of them every time I'm putting the phone back in its carry-case, usually the volume. The buttons at the bottom of the phone are also a bit random. There are four permanent buttons, plus a trackball. Unlike other users that I've read comments from, I don't find the trackball to be completely useless, because it lets you be more accurate about what you are selecting with the cursor. You wouldn't type with it, but it's not too bad for careful navigation or some games. That said, it does seem to be taking up some valuable real estate. There's a permanent "search" button which is basically pointless.

The voice recognition is far from perfect, but it's also much quicker than typing. If they can figure it out, then it will be a great input mechanism. Don't talk for too long or it will stop listening and abandon the attempt.

The keyboard is okay, and probably comparable to other phones. No comment really.

I haven't gotten Python up and running yet. I failed on the first attempt, but haven't started to analyse the problem, so I expect it will be working soon.

Anyway, that's my 2c!

'Python' vs 'python' @ python.org (Lexical Dispersion Plot)

2010-06-17T21:42:00.001+10:00

Something I love about mucking about with language are the neat features that seem interesting, but it's hard to figure out what, if anything they mean. I downloaded 50m of html files from http://www.python.org, took the text out and then loaded it into the nltk library. I then did a lexical dispersion plot on two keywords, "Python" and "python". A lexical dispersion plot basically shows where in a text file a word occurs. If the files were organised by time, for example, the left part of this plot (near x=0) would be the oldest text, and the right part would be the newest text. My files weren't organised along any meaningful vector, so the offset only really shows how the data clusters internally. Ideally I'd order the text files by last-modified dates so that we could do a time analysis on the database.

Clearly, "Python" is used far more often than "python" within the site. However, round near offset=0, there's clearly a cluster of "python" uses. Maybe there's one rogue wiki editor who just likes using the lower case. Maybe it's used in links. Who knows? It's clear though, that the Python Software Foundation is very active in its pursuit of proper capitalisation of its trademark!

I'd have some more stuff to post, but I got sidetracked by fine-tuning my wget arguments to get a better dataset...

Basic web-to-language-stats processing done

2010-06-16T22:35:00.000+10:00

Following process completed:
-- Use wget to capture a web corpus
-- Use BeautifulSoup to pull out the language bits (note, discovered nltk has some web scraping capabilities, need to compare)
-- Use nltk to read in plain text from web corpus
-- Print a few organisational language stats and plot a couple of neat graphs

Need to develop:
-- Automated downloading through code rather than manual wget stage
-- Possibly add PDF scraping
-- Store files so they can go into the text database organised by creation date/time
-- Use nltk to be time-aware so that time charts can be plotted
-- Clean up the class design so it's (more) elegant and useful

Need to document:
-- Technology stack involved

Possible future developments:
-- Build web page out of the information found
-- Try out on a few different organisations
-- Add some metadata / parameters, build some more interesting info
-- Extent capacity to show information where organisation is cited in standard news websites

Organisational language analysis tools

2010-06-15T23:32:00.000+10:00

Okay, so I'm just self-documenting some home hacking I'm doing at the moment to try to add some spice to my upcoming PyCon AU presentation. I thought I'd try to put together some code anyone could use to do some basic language analysis on their organisation. So far, I'm still building the language corpus which I'll be using. Tonight I've been using BeautifulSoup to parse web pages and extract the language components (defined for the time being as the string part of all the tags in a page).

This should be more than enough to create a plain text language corpus that I can throw at nltk in order to get some scatter plots of word frequency, maybe most-common trigrams, most-common sentences etc.

Step 1: Make a mirror of your organisation's website
Tool: Wget
Ease-of-use: Really easy

Step Two: Scrape out the language parts
Tool: Beautiful Soup (Python)
Ease-of-use: Straightforward

BeautifulSoup is currently pushing my laptop CPU up to 100% scraping 250m of web pages and seems to be doing a good job. I really love the description of the software on the home page:

You didn't write that awful page. You're just trying to get some data out of it. Right now, you don't really care what HTML is supposed to look like.

Neither does this parser.

For this kind of work, you really want something which is robust to minor quirks and hassles. The purpose is to get the most information for the least amount of work, and spending a lot of time polishing a data set is going to be wasted for most people's purposes.

Anyway, enough for now...

Facebook group for PyCon AU 2010

2010-05-31T22:31:00.000+10:00

PyCon AU Facebook Group

For those who are keen on Facebook, I've created a group for PyCon AU 2010, to be held in Sydney on the 26th and 27th of June (with the organiser's blessing of course). I'll paste in any even updates, and it would be great to get a good number of people joined up in order to increase the visibility of the event.

Yes, you too can be part of the message that PyCon AU is going to be absolutely fantastic. Please leave a comment on the wall and let us know what you're looking forward to.

Cheers,
-Tennessee

Creative Edge, Toronto: Article on Listening

2010-03-30T20:28:00.000+11:00

Readme...

Creative Edge, Toronto: Article on Listening

"My old pal Harry and I are walking in the park, improvising like two jazz musicians - except we're playing with words not melodies. He throws out a line. I have a comeback. He does a riff on my response. Pretty soon we're laughing so hard we're crying. Eventually we collapse, exhausted, on a park bench.

"That was amazing, Harry," I say, "Why don't we do it more often?".

I'm just being social, not really expecting a response, but Harry takes my question seriously. He leans closer, lowering his voice, like he's confiding in me."

Incoherent Ramblings on Imagination, Semantic Nets, AI

2010-03-18T00:45:00.000+11:00

I have been reading my AI textbook again, plus Daniel Dennett's frankly *brilliant* book "Darwin's Dangerous Idea" and I'm afraid it has sparked my overactive imagination!! If you read one serious book this year, make it that one.

Suppose you tried to build an AI entity suchly...

Suppose you start with a network of nodes. Each node is either an axiom node or a composite node. An axiom node is a node which represents something believed to be absolutely true, and un-decomposable at the current level of abstraction. A composite node is one which is not an axiom node. Composite nodes may be grounded (derived from only axiom nodes and other composite nodes) or un-grounded (only partially derived from axiom nodes and other composite nodes). The 'somethings' in the nodes may be facts as we know them (propositions about the objective world, say) or may be undescribed non-propositional nodes which are a part of learned relationships.

One could "imagine" things by overlaying a supposition network over the knowledge network, perhaps using an inheritance structure (i.e. the imagined world derives from the real world but some subnetworks truth propositions are deemed to be different).

Input streams and output streams are used to embody the agent. As such, the agent is constantly writing new observation nodes into its knowledge network.

Over this knowledge network, rule processes run. These rule processes are also stored in the knowledge network, but are marked as rule processes. A simple rule process would be one which could enforce simple truth relationships. (i.e. If A --> B and A is true, then make B true).

Nodes would be scanned for identity (Node A is Node B if node A and node B are sufficiently indistinguishable with high confidence) representing network simplification.

More complex rule processes could be "imagined" and run over the imagination network to evaluate their performance. In this way, imagination allows the entity to test potential rules for their truth value.

The model scales to the extent that the network can be decomposed (i.e. the 'locality' factor of the network with respect to the problem at hand). For example, the application of simple rules which operate on say 3 nodes would be parallelisable and scale well, while complex rules which say operates on all nodes to a depth of N and branching factor of B would not parallelise as effectively.

The fundamental mode of processing would be a build-and-test model of imagination where new knowledge networks are imagined then evaluated, and then incorporated if they perform better. This makes the system a memetic evolutionary system, which seems to me to be a prerequisite for sustainable machine learning.

Learning success comes through untrained re-enforcement based on assessment of expectations of future observational input. Because the entity is based on input *streams* and output *streams*, then even non-action is a choice of the agent.

All concepts, then, are either grounded concepts (fully understood and linked to base phenomena) or ungrounded concepts (not fully decomposable). I would also suggest that truth generally be a real-varying amount, but that Absolutely False and Absolutely True also be allowable values.

This model supports "making assumptions" by building an imagination network where, for example, all things which are believed to be 98% true are taken to be absolutely true, then the results evaluated for performance. Experiments can be run in which nodes can be tested for unification, or a node could be split into two nodes then learning and assessment re-performed.

Of course, I haven't even addressed any questions about the initial state of the system, how the learning algorithms actually work (how is knowledge propagated and how is re-enforcement applied), whether the AI needs multiple conceptual subnets, how current standard problem-solving techniques might be integrated etc etc. But that's okay, this is my blog and I'm just rambling.

Cheers,
-T

Resolver Spreadsheet = Awesome (ish)

2009-11-13T17:08:00.000+11:00

Here are some reasons why Resolver Spreadsheet is awesome:

1. It produces Python code. There is no particular reason why I couldn't virtually cut-and-paste something from the spreadsheet as an algorithm in my application. Vice versa, there is no reason I couldn't "prove" an algorithm to a co-worker or supervisor by setting up some sample runs and graphing them.

2. It produces Python code. You can edit it to change the semantics. If you like coding, you can do it!

Things that are not as awesome:

1. It only runs on Windows

2. There's a 31 day evaluation. I understand why, but I still find evaluation periods annoying and too short. If I could have a 6-month evaluation period, then I'd be happy.

py3k fail

2009-09-24T20:26:00.003+10:00

Okay, it's not py3k that's broken. It's just that I can't load jpgs (I think) because I can't use Pil to create a TkSomething graphics canvas (at least that's how it goes in the tutorial). Which means I can't try out what I wanted to try out using py3k. Hum.

So I've reconfigured the project to Py26. It's a bit disappointing not to be able to use the latest version of my favourite language, but at least I can use the most exciting language features in Py26.

-T

Really Powerful AI: how close is YOUR system?

2009-08-27T15:39:00.003+10:00

I read a lot about AI. Many people think AI is, one day, possibly soon, going to achieve a level of competency such that it will revolutionise the planet and destabilise humanity as the only intelligent force. Some people point to a number of identifiable trends in computing, usually trends in computing power, and claim this indicates a progression towards that stage.

Some such people are crackpots, while others are insightful individuals with the knowledge and experience that they should know what they are talking about -- certainly well enough to give a valid and sensible opinion.

However, I suspect that many people aren't directly working on a system which demonstrates such capability or promise.

So here's my question to anyone working in AI (or any related area). How close is the system that you personally work on, right now, to demonstrating the kind of ability which would contribute significantly towards an independent, thinking machine?

Cheers,
-T

A few days of WingIDE; a few years of Eclipse

2009-08-21T10:10:00.002+10:00

I thought I would just quickly blog about my experience giving Wing IDE a fair trial, using it exclusively to fix one particular bug in my system. The short version: it's okay. It needs a visual diff.

Okay, so now for the long version. My application is a hybrid Python/C application, with my particular turf being almost exclusively in the Python code. The team has basically adopted Eclipse + PyDev as a kind of default environment, but it's not proscriptive. Eclipse works, there's no doubt about it. It supports pretty much the workflow I want, does a reasonable job of editing, and a reasonable job of communicating with our SVN repository. It's flexible, so I can do C work in it, hook into our ant targets, change the repository and manage multiple repositories. It's popular, so there is a wealth of information on teh internets. On the downside, I find that it's slow, the code folding is flaky (try folding a file, then editing just one method), prone to crashes.

I've long known that Wing IDE exists, and for extra kudos it's written in Python. It seems to be a genuine competitor, and I thought I'd take it for a spin.

The first thing I had to get used to was how it treats projects. It's probably better actually, but different. It creates project files, in which it stores (apparently) a list of files and directories which are included in the project, plus (presumably) various project settings and configuration files. However, it doesn't actually make your directories for you. You have to do that yourself, either during 'save as', or externally and add them to the project. Well, okay, I'll do that. But what's weird is that there's no capacity (that I could find) to create a project from a code repository. Checking the code out is legwork you have to do yourself, before you create the Wing IDE project. Well, okay...

So I did that. Once the project is created, and the directory added, you can then enable version control for the project and then you finally get some GUI help with your repository. However, this is basically where Wing IDE starts to fall down, and unfortunately it's one of the things I ran into first. So, I'll hold off on criticizing this functionality while I talk about the good points.

It's quick and responsive -- much nicer than Eclipse for doing the job of typing out code. Wing IDE is definitely a nicer editing experience.

Its folding is mostly better. It doesn't randomly unfold while you're typing, which is a big win. Unfortunately (for my tastes) it folds all the methods directly next to eachother so there's no whitespace in between. I don't know if I like that, and I'm definitely not used to it. I prefer a gap so I can start typing in between two methods if I want to. But it's a big win over Eclipse.

Its two-pane view is a little better, and again way faster. Eclipse throws a shoe sometimes when you have the same file open in two panes. It would be neat if I could flow one file into a kind of two-column layout with integrated scrolling, but Eclipse doesn't do that either.

Its search is better and easier to use.

In short, if it weren't for the code repository issues, to which I will now return, I'd much prefer Eclipse. So now for the show-stoppers.

In Eclipse, as soon as I change a file, the project browser clearly marks that I have diverged from the latest repository version. Wing doesn't.

In Eclipse, there is a *great* visual diff tool. Wing IDE will just show you the text diff file.

Eclipse has a built-in repository browser. Wing doesn't.

Anyway, I've run out of time to write down any more thoughts, but that was my experience trying out Wing. It needs to be more capable with code repositories before its functionally at the same level as Eclipse. When it gets there, its speed will put it in first place (for me).

-T

I know I should know how to do this, but...

2009-07-28T10:06:00.002+10:00

I'm convinced I have read about why not to do what I'm in the middle of doing. It even feels like a bad idea, but I'm having a brain failure. I'm refactoring. I have a bunch of methods which basically take a list of say 20 similar objects, and merge the similar objects together into new objects. Imagine having a list of 20 numbers. Any numbers within 1 of eachother are 'similar', and should be removed from the list, then their average should be added back to the list. After the process, you might end up with say 4 'representative' numbers.

Okay, now I'm not dealing with numbers, but with complex objects, but the principle is the same. Rather than having like 6 different methods doing basically the same thing, I thought I would implement just *one* mergeList method which would take a pairwise mergeMethod which would then get applied to each pair. Okay, great. So I did that, but now I've realised that I'm going to end up with like 6 different pairwise mergeMethods, when really most of *those* are still pretty similar. In fact, some of them are already in my codebase and are written such that they take a modal switch as an argument.

So now I'm in this situation where I have a generic merge process, but I need to pass modal arguments down to the pairwise merge method. Now for the evil bit.

So, there's this thing called **kwargs. I can write my generic merge process such that it will accept arbitrary keyword arguments, which it can then pass directly to the pairwise merge methods. I could then call my generic method with a pairwise mergeMethod and additional arguments to be passed to that mergeMethod.

Is that evil? Should I be using inheritance instead of modal arguments?

Also, in self-defence, a lot of the constraints here come from dealing with a mature codebase. I'm just trying to work out where I should decide to draw the line and get things Done.

Cheers,
-T

HELP: How do you do Test Driven Design and Prototyping?

2009-07-03T09:22:00.003+10:00

Here's where I fall over with TDD. Let's imagine a standard day in my life...

I have some programming problem. I need to build a Thingy to do Stuff. I don't already have anything that does something similar, so I sit down and think about the problem. Along the way, I figure out some approaches to the problem. I don't really believe in BDUF, so mostly I'll just start coding. This kind of exploration is what helps me think, and so I'll build 2 or 3 partial programs before I start to converge on something approaching a solution. Let's dot-point the process so far:

* Problem. Solution?
* Analyse
* Maybe scrawl out a flowchart
* Write a program that for some simple input, generates something like the right output
* Gather up more input data sets, and pump them through the program, extending and fixing as I go
* Reach workable solution

Okay, now a few background points. This isn't how I'd approach a big, team project. But it's how I approach anything I have to solve by myself. I can't just navel-gaze and come up with a great program design, and if we're being honest, I'll bet you can't either. To reach a decent design, I basically need to build 2 or 3 mediocre attempts first.

Now, as far as I understand it, TDD goes hand in hand with unit testing, which is all about small, well-tested, re-usable components. Well, that's great if your fundemental starting point as a designer / developer is the component. But really, it's not. Your starting point is the problem, and the process is one of decomposition and analysis.

Some problems lend themselves to an easy decomposition. A problem which lends itself to a decomposition will immediately make you think "hey, I know how to solve this. If only I had a sorter, a comparison algorithm, some kind of message generator and an input parser this would be a cakewalk!". That kind of problem isn't so hard, and is made out of nice, well-defined objects whose role is well-understood.

Other problems make you think "uh-oh. This one's going to take some coffee, a whiteboard and a fair bit of muttering." Some part of me thinks that the better and more experienced you get, the more new problems should tend to fall into the first category, but in fact I just tend to get given harder and harder problems (or so I think!)

So this is a question to TDD experts. What is the design process that should be followed when confronted with a new problem?

Cheers,
-Tennessee

Call for interest: should I run a python AI competition?

2009-06-30T15:29:00.003+10:00

This is just a shoutout to see whether there would be any interest in my running a Python AI competition? There are a few Pythoneers who are into AI that I know of, and it occurred to me that one thing which could be done to serve the community would be to offer a competition.

At this stage, I'm thinking either of challenges which relate to basic AI algorithms, or perhaps building a chatbot/twitterbot. The first challenge should probably have a low barrier to entry, so perhaps I will put together a multi-stage or multi-challenge competition so that people can choose their own level of competition.

Leave a comment with (+0) if you think it would be neat, (+1) if you would take part, (-0) if you're not really in favour, or (-1) if you would sabotage the competition :)

Cheers,
-T

Python, running in a browser

2009-06-18T09:48:00.001+10:00

http://lackingrhoticity.blogspot.com/2009/06/python-standard-library-in-native.html

I have a dream: one day, I will be able to not know Javascript. Thanks Mark!

-T

Driving unit test takeup with code coverage

2009-06-17T13:37:00.002+10:00

For anyone who has not fully gotten organised with unit testing and code coverage, this is for you! :) My project involved a large inherited codebase which has good black-box testing, but little unit testing and no coverage metrics. Tackling unit testing was always something impossible for me -- how do you take 96, 000 lines of Python and 1, 000, 000 lines of C, and build unit tests? (nb the lines-of-C count seems high, but it's what 'wc' said.)

The general advice is not to try -- but to get traction by just writing one unit test for the next bit of new code you write. Then, the next bit, etc. Eventually, you will have figured out unit testing and will then make an appropriate judgment on what to do about the body on untested code.

I have typically found this to be quite a large hill to climb. I work only on a subset of the code, which practically requires me to invoke most of the code just to get to my part. Most of my methods require such a lot of setup that it seemed quite infeasible to tackle unit testing without doing some more thinking about how to do this in a sane way. Setting up and tearing down my application was just not going to be feasible if I were going to put a lot of unit tests in place -- I reckon the setup would have cost between 1 and 7 minutes per test!

This got relegated to the too-hard basket, and set aside until now. Here's how I found a way to get traction.

What turns out to be pretty straightforward, is integrating coverage testing. You more-or-less just switch it on, and it will record to a file across multiple program invocations. This can be used to count coverage across test scripts, or user testing, or development mode use, or indeed in operations.

I ran this through about half my black-box tests, and found I was sitting at around 62% code coverage. That's not too shabby, I reckon! I know for a fact that there are quite large chunks of code which contain functionality which is not a part of any operational code path, but is preserved for possible future needs. I estimate 25% of our code would fall into that category, lifting the coverage to 87% for the sub-area of code which I work on. Now I've got numbers that look like a challenge, rather than an insoluble problem!

I think that's the key... make lifting code metrics an achievable challenge, then it will seem more attractive. It's probably important not to target a particular number. I know '100% or bust' may be what some advocates would put forward, but for anyone new to the concept, I personally feel that simply measuring the current coverage, then understanding where that number comes from and what it means, it the more important achievement.

What is clear is that I'm not going to easily lift my coverage metrics beyond a certain point simply through tactical black-box testing. I'm going to have to write tests which go through the code paths which aren't part of the operational configuration. I'm going to have to write tests with very specific setup conditions in order to get to lines of code which are designed to handle very specific conditions. All of a sudden, I've got achievable and well-defined goals for unit testing.

I call that a win!

Cheers,
-T

Anyone been to an Australasian Joint Conference on Artificial Intelligence

2009-06-17T12:58:00.001+10:00

Dear lazyweb,

Has anyone been to one of these conferences in the past? Did you find it valuable as an attendee? Please feel free to email me directly at tleeuwenburg@gmail.com if you would prefer not to leave your comments publicly.

Cheers,
-T

The Pinball Principle

2009-06-11T14:36:00.003+10:00

I read a fair bit about different development methods. Agile is pretty popular amongst developer advocates, but I would argue almost everyone in charge of development *actually* just runs with a grab-bag of "what works for them" taken from Waterfall, Agile, what their peers think, what happened on the last project etc. They'd be mad not to -- can you imagine running development based on what doesn't work for you?

Some people talk about bottom-up vs top-down, inversion of control, directed vs undirected, the importance of use cases, test-driven development etc. Usually, the argument is that if you tick whatever list is being advocated, better software will get written in an efficient amount of time and deliver what was ordered to the greatest extent. But, somehow, it never seems to work out that way.

There is a huge complicating factor, which is that most development activity is done due to an obligation owed by an employee to an employer, client or other stakeholder. The developer ends up responsible for not just managing their own development practises, but managing their clients through the process also. If there are just a few stakeholders, then that's easier (unless they're very difficult stakeholders). Issues come up when there are multiple stakeholders, and when there is conflict between those stakeholders. The best possible course of action ceases to be "build and deliver approximately X" and becomes "make all these people happy to the extent possible".

How does making people happy fit in with SLDC practises? Where do you stand firm, and where are you flexible? When do you tell your manager "we shouldn't do that" and when do you just fit in? How do you sell a use-case Agile practise to a control-oriented manager? What are the ethics of accepting your manager's control-oriented activities if you think things could be done better another way? When do you take a decision on your own shoulders? How do you actually adopt new design practises? How does it fit into the development team and the organisational culture?

Most people in an organisation run according to what I call the 'pinball principle', which is to say they get batted around by their key stakeholders, bouncing off obstacles, temporary victories, other groups etc like the objects in a pinball game. To some extent you can control your own fate and direction, but there are a *lot* of things which you can't control.

In a large development team, a kind of 'surface' will form around the team, representing that team's culture, which can protect those inside of it. However, some people either don't have a protective surface, or they are right on the edge of it. In these cases, you can't simply adopt a practise and hope it will result in better outcomes, you need to use personal insight to negotiate the situation you find yourself in, and no amount of 'best practise' will prevent your key stakeholders looking you in the eye and demanding action, their way.

The question is whether there is anything we can learn from the pinball principle. If our environment is so complex and demanding, is there anything about that which should inform our behaviour? Should we acquiesce to all incoming requests, devolving responsibility, and concentrating instead on doing the best job we can? Or will something like the Agile methodology allow us to write great componentry which will then turn out to be just the thing for the next request? Should we attempt to educate stakeholders in appropriate negotiation processes, or should we instead rely on meetings and face2faces to take care of things. What is the cost of attempting to enforce process?

If project direction is likely to be set by pinball mechanics, how can we best write software knowing that will happen?

I don't know the answers -- not all of them anyway. But here is *my* list of things which I think will help keep the development *direction* on-track, and best serve long-term development success. I have just come up with them now, but they are based on ideas which I have been refining for some time now. I'd love feedback on the list and this post!

1. Get all of the stakeholders into the same room at least once a month. Make them review their priorities in front of everyone else, and get agreement on the priorities for the development team.
2. Maintain a whiteboard or wiki with the top four development priorities. Focus prioritisation on the top four issues.
3. Email out a bulletin every fortnight without fail to all stakeholders and users who are interested, including the current priority list and progress achieved.
4. Hold 'air your grievances' sessions regularly (say twice a year), especially if there is a layer of bureaucracy between you and your end users. Don't minute them.
5. Identify what makes your stakeholders actually happy (i.e. less user complaint, positive feedback from other departments etc) rather what what they say they want (more progress for less money). You will only make other people happy by reducing the pressure they get from the pinball board. That's only sometimes the same as what they're asking you to do
6. Pursue a two-layer development methodology. Make the bottom layer whatever you like (Agile, TDD, Waterfall), and make sure the developers understand it. Keep on top of new ideas here. Make the top layer As Simple As Humanly Possible. This is your 'interface' process, and keeping it simple is the only way that others will follow it. If you can't explain it at the beginning of every meeting in two slides, then it's too complicated.
7. Document Everything.
8. Publishing = profile = perception. Make sure you nail the following items: project web page, prototype web page, email bulletins, conference/journal publications (if relevant), regular meetings/presentations

Regards,
-Tennessee

Multi-party vidoeconferencing

2009-05-29T11:09:00.002+10:00

Is it practical yet to have a multi-party videoconferences? What free software should I use? What will work for both Windows and Linux?

Cobra -- Next Big Thing (in a few years time)

2009-05-19T09:45:00.002+10:00

Thanks to Simon Wittber for alerting me to this language via his blog post, "Cobra vs Python".

The Cobra homepage is available here.
The Cobra v Python overview is available here.

I'm going to make the ludicrously early call that this language will be the next big thing, eventually. I will not be moving away from Python any time soon, since (a) my job depends on it, and (b) it's fully operational now. However, I really do think that Cobra is, if not simply 'better', an important step forward in the evolution of dynamic language generally. Watching Cobra develop will provide insights for all languages, I have no doubt!

It has the features that make Python great, plus important features from other languages which will make it even more popular and even more palatable. It allows the kind of productivity you can only get from being able to 'throw something together' using elegant syntax and high-level semantics with dynamic typing, but then it *also* allows optional static typing to get the most out of compile-time checking to provide early-warning of type errors and an extra layer of guarantee of program functionality. It also includes contracts, which, while they could in general terms be implemented in any language using a lot of assert statements, are supported by Cobra with a neat syntax which will encourage their use. I can well imagine using this as a strong argument in its favour -- developers could prototype a system or server using dynamic typing, then go back and tighten up the screws with static typing as appropriate and the introduction of contracts.

It deploys to all major platforms, including .NET.

For me personally, the best features are:
* Decimal arithmetic by default (i.e. the literal 5 will become a decimal, not a float)
* Optional static typing
* Contracts
* That all the best features (for me) of Python are included

Cheers,
-Tennessee

Two big thumbs up for OpenGoo

2009-05-14T11:45:00.001+10:00

http://www.opengoo.org/

It's like Google Docs on steroids, but open-source and installable on your intranet right now.

I've only spent about 3 minutes exploring this software, but it appears to be absolutely amazing.

Two thumbs up, OpenGoo!!!!

Pro bono activities

2009-04-27T13:50:00.002+10:00

Do any IT firms have explicit pro bono programs or open-source contribution programs which might mirror the pro bono activities of legal firms? The law industry obviously has a long-standing culture of engaging in pro-bono activities with the imprimatur of the corporation.

A few IT companies obviously house and/or champion open-source projects. However I'm not sure how these are really structured. Does 'the system' adequetly encourage and/or support these activities (beyond simple competitive-self interest, or enlighted corporate social responsibility)? Are there tax breaks for companies engaging in open-source or pro-bono activities such as ethical or charitable software development?

I just don't know. Perhaps the answer is very well-known, but I didn't turn up much in 20 minutes of looking. I just felt like blogging my curiosity on the topic...

-T

Ramblings on AI

2009-04-09T12:26:00.003+10:00

As some know, I work on a natural language generation system for weather forecast reporting. I also have an interest in general AI. This post represents little more than the idle thinkings from my lunch break...

The ultimate question of AI is, of course, can we build a machine that thinks (for various definitions of thinks)? One response to that question by anyone interested in programming is to imagine how such a thing might be constructed.

I have presented on the topic of the structure of an AI agent before. I have been trying to flesh out some aspects of what many might call the most fundamental component, the reasoning and learning engine. As I write that, however, it is of course possible that these be two separate systems, but I will continue as though they were one. By learning here, I don't just mean laying down and accessing memories, but rather the act of mapping a new situation onto an existing conceptual framework, either by creating new conceptual substructures or recognising the applicability of an existing conceptual substructure.

Along the way, a number of people have offered their insights as to what may be the best way to perform reasoning and learning, but I'm not convinced they have solved the issue of conceptual structure yet.

I have come to think that any AI system will need, in addition to generalised learning tools, a number of pre-written or pre-constructed concepts and processes for understanding information. An example of a conceptual construct may be a Bayesian Network. The agent might go -- oh hey look, here's a situation with a bunch of discrete inputs which I can recognise, and a few output states! Great, I can map this new situation onto a Bayesian Net and learn some appropriate responses.

Going further, the system may be able to recognise a new situation as being related to a known situation. Oh great -- this situation is really a variation on the Poker Card Game! I'll copy that network and get a head start on my learning. I'll just set these probabilities to .5 since they're unknowns and away I go.

However, what's not clear to me is how an agent could possibly going about choosing the appropriate input schema, come up with its own output states, or infer network structure. This problem extends to many kinds of reasoning engine -- rule-based systems, ANN, others.

Clearly, there is no One True Network to rule them all. I don't know whether anyone can conceive of a any structure which is inherently able to perform all thinking. The human brain doesn't appear to be built that way either. To my understanding, it is born with some inherent structures plus the ability to learn. It demonstrated remarkable plasticity and regenerative capacity, but it's still the case that there are certain physical areas which strongly tend to be responsible for particular kinds of thinking.

It is also probably true that no-one is every going to have the kind of direct insight and 'just come up with' a fully generalisable reasoning engine capable of learning how to deal with any situation.

However, it does seem to me to be possible to proceed along the following path:
* Identify some situations
* Write conceptual structures which can reason about those situations
* Write additional software which attempts to map new situations onto existing situations
* Write software which is capable of evolving new conceptual structures to some extent

It seems as though learning new things needs a structure to cling to -- like evolution. It's very difficult to cross certain functional divides solely through a process of evolution. Unlike organisms, who live, reproduce and die, conceptual structures can do more than that. We can use our insight as humans to build more advanced structures more quickly than may arise through chance. If we see the mind itself as consisting of a low-level, always-on, processing algorithm in which a multitude of conceptual structures exist, I think that could help. We should be able to give any AI a head-start by building some specific conceptual structures while still allowing others to evolve and grow.

...

well, I'll think more about that later.
-T

Call a function recursively

2009-04-03T16:25:00.002+11:00

At the risk of public embarassment, I didn't actually know I could do this:

>>> def recursive(list):
... if len(list) == 1:
... print list[0]
... else:
... recursive([list[0]])
... recursive(list[1:])
...
>>> recursive('hello')
h
e
l
l
o

It seems perfectly obvious now -- it's not like I've never written code like this in other languages, but for some reason I just hadn't been thinking about it.

Open Science: Good for Research, Good for Researchers

2009-04-02T09:27:00.001+11:00

http://scholcomm.columbia.edu/open-science-good-research-good-researchers

Quote: '''
Open science refers to information-sharing among researchers and encompasses a number of initiatives to remove access barriers to data and published papers, and to use digital technology to more efficiently disseminate research results. Advocates for this approach argue that openly sharing information among researchers is fundamental to good science, speeds the progress of research, and increases recognition of researchers. Panelists: Jean-Claude Bradley, Associate Professor of Chemistry and Coordinator of E-Learning for the School of Arts and Sciences at Drexel University; Barry Canton, founder of Gingko BioWorks and the OpenWetWare wiki, an online community of life science researchers committed to open science that has over 5,300 users; Bora Zivkovic, Online Discussion Expert for the Public Library of Science
'''

Call for Reviewers to The Python Papers

2009-03-27T16:22:00.002+11:00

Hi all,

This is a call for assistance from the Python community to register as a potential reviewer for The Python Papers. We are currently unable to process our academic articles as effectively as we would like due to a small list of reviewers. It would be wonderful to have some more people register with ojs.pythonpapers.org and flag their availability as reviewers for our academic stream. You can flag this in the "reviewing interests" section of your profile.

Reviewing a paper need not be intimidating or difficult, and is a great opportunity to contribute back towards the community in the form of supporting academic work which involves Python.

For those who do not feel they have an academic bent, we also make use of our list of registered users when exploring articles in the technical stream.

Thanks very much,
-Tennessee Leeuwenburg
(Co-Chief Editor, The Python Papers)

Wheelchair tech

2009-03-13T19:38:00.003+11:00

I was walking about my city recently, and observed a woman in a wheelchair using some form of touch-screen interface, on a screen which was integrated to the chair. Further to the mobility difficulties, she appeared to suffer from a more generalised difficulty with effective use of her limbs. I do not know what she was doing; nor what the functionality of the interface was. Here is what I think *should* have been on it. What do you think?

1.) Integrated GPS system displaying directions overlaid on a map
2.) A large-buttoned interface tied into a telephone dialing system, allowing ease-of-access to Taxi services, personal contacts and any relevant emergency services that may be required
3.) If necessary for safety, a GPS tracking device. For some individuals with a carer who is partially responsible for their wellbeing, the capacity for that carer to easily contact and locate the impaired individual could substantially increase freedom of travel
4.) Additional geographic information systems covering accessibility of local public transport; accessible footpaths and eating establishments; accessible public toilets and other facilities

Any other suggestions for this somewhat naive list?

Watermarks, signatures and security

2009-01-02T13:39:00.002+11:00

I just had this idea... I was thinking about computerising organisaitonal processes which require an approval signature. Suppose manager X needs to approve request R. If he has a drawing tablet, he could digitally sign a document -- that is to say associate it with the document either by overlaying it or attaching it. However, that's somewhat susceptible to hacking by taking an image of a signature and associating it with some other document. That is to say, it's hard to guarantee that the associated signature is 100% confirmed on the document being signed.

Supposing, instead, that I had software which did this:
1.) Take signature input from drawing pad
2.) Generate a hash from the signature image
3.) Use the hash as a salt for a watermark
4.) Create a watermarked document
5.) Overlay the signature image onto the watermarked document image
6.) Lodge the image hash in some collision detection database

It seems to me like I should be able to encode the signature hash into the watermark document somehow. Then, it seems like I should be able to guarantee that the watermarked document was created using the signature that's on the document.

A forger/hacker could not then take the signature and do the same thing to another document, because the duplicate hash would then get picked up by the collision detection database. Further, each colliding document could be identified by its watermark and signature hash. All anyone using the system need to is check that the document is watermarked. The system guarantees that all watermarked documents are properly authorised.

Should a *real* hash collision occur, the authorising supervisor could just re-sign the document.

This relies on every signature being subtly different. That is, the tablet must have quite high resolution to capture the slight differences of pen pressure and letter shape in each signature. This doesn't seem beyond the realms of possibility however.

And, once all it said and done, there exists a system where:
* The approving manager can just sign digital documents completely straightforwardly
* All authorised documents are stored and tracked
* All digitally watermarked documents have a full history

What does the blogosphere think?

bluehackers.org

2008-12-15T09:52:00.002+11:00

"It started with Arjen doing the last lightning talk at OSDC 2008… a
quick show of hands on who else had dealt with (or was dealing with)
depression. Everybody had a look around, and thus knew that they
weren’t alone. Afterwards, there was more positive feedback which
continued over email in the days that followed. Someone suggested
starting a group, and the same day bluehackers.org was born.

The objective of this initiative is to make visible that there are
many fellow geeks among us who are intimately familiar with
depression, anxiety, or bipolar disorder. It helps to know you’re not
alone. And it’s not because we’re geeks, but because we’re human. The
Australian BeyondBlue site is of course an excellent resource, but,
because geeks have a specific work environment, there are also
particular challenges in dealing with these issues, and that’s where
we feel our group can help with additional insights, tips, and posts
from others with experience.

Using the logo, we can also make the topic visible at meetings and
conferences around the world, ensuring that indeed no geek need feel
alone in this, or feel unsupported. They can simply look around and
see. Anybody will be able to show their support and understanding, in
a kind and non-intrusive manner."

...a repeated message from Arjen Lentz

OSDC liveblogging: OLPC in Australia

2008-12-05T16:14:00.002+11:00

Presentation by Pia Waugh, OLPC
pia@olpcfriends.org

To clarify:
- project is not the KR thing
- not paid by OLPC, voluntary talk

Great FOSS project
Opportunity to create a regional approach
- connected classroom
- shared resources
- supplementing poorer countries
- do have great need in our regions, despite relative advantage overall
- create community of educators etc

Imagine if many people's education were based on an open source platform

Child takes a sense of ownership / responsibility for OLPC device
Better for each to have their own rather than to share for this reason
Target group is 6-12 years old

Important tenet is connectivity
- Can visualise mesh network of who is nearby
- FOSS, building blocks, building blocks for creativity

Some of the technology
----------------------

Antennae takes 5V in, can be used independently for mesh range estension
School server version
- jabber, other pooling tech, security tech, alerts if laptops leaves it will shut down unless it can reconnect to school server after some period of time
- backs up other laptops, allows teacher monitoring etc
- mic jack doubles as voltmeter

Regional Vision
---------------
- Great need in the region
- Build a strong developer and educator community
- e.g. indigenous communities and children in impoverished areas

5000 laptops donated to regional project, trials include many pacific island communities
Providing sat networking to some of the most remote areas
Some potential to travel with OLPC and contribute in the community, volunteers needed
wiki.laptop.org/go/OLPC_Oceania

Consideration of cultural values
--------------------------------

When rolling out, most pacific culture has healthy respect for teacher as the leader

Try to make sure teachers get the device early, extensive trainining, teacher is the person that gives the children the resource rather than from random externals.

Fit with what groups are already trying to achieve with their schools

Australia's first trial is happening
------------------------------------

A single laptop by itself doesn't have so much value, need classroom environment
e.g. spelling tests, collaborative writing activity

World's first remote-connected classroom. That is, the local teacher is connected to remote classrooms, video chat etc.

Video demo of remote classroom showing flashcard spelling exercises with student

Observed benefits to children
-----------------------------
Educational:
- Literacy improvements
- ICT skills -- typing, programming, games
- Maths
- Collaboration

Behavioural:
- Engagement of troubled children
- Reduced truancy

Example of Python shell where you just run demo code then change it yourself as an exercise

Example of a particular troubled kid from a very disadvantaged background who engaged through the laptop environment, partly because he could engage with his interest area, partly because the laptop was his, that he could use according to his own preferences and progress in his own time.

Benefits of a regional process:
-------------------------------
Funding, support, access to volunteers, knowledge xfer

OSDC liveblogging: Your code sucks and I hate you

2008-12-05T14:08:00.002+11:00

Your code sucks and I hate you

First time patch submitted to OS project:
- nerve-wracking experience, judgement
- having people look at your code, give feedback etc is very powerful and important, best tool we have to get better as programmers

What's a review?
- Anyone even just talking about code
- A few types
- Involuntary review
- Random inspection
- Post-merge
- Pre-trunk (really good way of working, how bzr works)
- Pre-merge is the new hotness
- Enforces conformity (style consistent, ensures quality, enhance clarity)
- Moves from 1 person who understands code to 2 people

A few things in code review than can cause social problems
- Being told your entire approach is wrong
- Better to have a conversation before coding starts
- While you're there, totally do something else also
* need to be allowed to fix small incremental issues
* unclear outcomes (ie "this is bad", not what about it is bad)
- Name-calling is bad, doesn't help, talk about code not people
- Confusing personal preference with objective worth
- Filibustering -- what happens when you are outnumbered, but can win an argument by talking lots
- Lag is bad, reviews need to be fast, lag builds up
- Need to know who are the reviewers

Good things (how to do good reviews)
- Specific feedback
- Be thankful for code submits
- Ask questions where you don't understand stuff
- Before you start coding, talk to someone about it. Anyone.
- Decide to who gets the final say to head off future conflicts

Questions:
- Worst have experienced professionally is "that's a stupid idea"
- Language barriers can be an issue
- What happens if there are not enough developers for review?
- Generally in favour of doing code reviews in public or opt-out forum
- Be specific regarding thanking people also i.e. "I'm looking forward to using this feature"

OSDC liveblogging: Natural language generation in weather forecasting

2008-12-05T13:34:00.001+11:00

Yesterday, I was unwell so I didn't keep up with my presentation blogging. Here is my presentation on Natural language generation in weather forecasting.

OSDC liveblogging: Legal Issues in OS

2008-12-05T11:35:00.003+11:00

Brendan Scott, OS lawyer based in Sydney. (opensourcelaw.biz)
[n.b. this is particular to Australian law]

* Some analogy about driving to a location with a bit of suboptimal route planning. Many decisions along the way.

* When engaging a lawyer, this is like reading a map before setting out
* Not always necessary / justified, this depends on a bunch of things
- familiar territory?
- time and cost?

Hypothetical business growth path:
- start
- build it up
- sell it

Setting up an incorporated vehicle makes things (significantly) easier
The word "partnership" has a specific legal meaning, including:
- you are personally liable for your partners actions in the business
- best to avoid unless want to be in this specific relationship

How would you react to someone taking code from a project you are working on?
- If you have good record-keeping, can sort out rights relatively easily, even if you don't have a copyright record. Just need records of activity and can bootstrap copyright attribution

Contracts take effect at the time of signing

The conduct of a negotiation limit the things you can say in the future to try and vary the contract -- i.e. major points have been ventilated. Difficult to bring lawyer in at this stage to make nontrivial changes.

What else might a lawyer do for you?
- Dispute Resolution. Often a case of identifying what the problem is and identifying win-win or least-loss outcomes
- Sometimes knowledge of the law is extremely relevant

OSDC liveblogging: The State of Python

2008-12-05T11:16:00.002+11:00

Presentation by Anthony Baxter

* 2.5.2 changes came from App Engine
* The Big News
- 3.0 is done
- .X releases are backward-compatible
- 2.6.1 almost out
- tinyurl.com/python3

IronPython continues on merry way
* Really fast
* Visual Studio is great
* First major bit of software with a decent license (pretty much Berkely)
* Targets 2.5
* Mostly works on Mono

Upgrade code:

* Take old code
* Port to 2.6
* Use -3 flag to show warnings
* Fix those
* Then run 2to3 -- rewrites code to 3 compat
* django is out for 3

Demo of upgrade to small app and highlight some specific changes
* print is a function
* print("foo", end="")
* __repr__ --> repr builtin, backticks gone

What's the future?
------------------
2.6.1 soon, 2.5 final not far off
3.0.1 a few months away

d.items() returns a dictionary view

If you want to learn python, learn in Python 3. New book coming out soon from someone who has been tracking python 3 for some time.

OSDC liveblogging: Bazaar (VCS)

2008-12-04T12:03:00.002+11:00

Gentle introduction to version control using Bazaar
Michael Hudson from Canonical

What is Bazaar?
Distributed VCS
No privileged sentral location
aims to be safe, friendly, free and fast

'bzr init'
'bzr add'
'bzr ci -m "initiam import"'

'bzr push' uploads or updates branch data on the server
'bzr branch' - get a copy of a remote branch
'bzr pull'
'bzr merge'

Some concepts
-------------

WorkingTree - collection of version controlled files and directories
Revision - A snapshot of a working tree and the fundamental object in bazaar
Branch - An ordered series of revisions
Repository - A place where revisions are stored
Commit - create a new snapshot/revision
Delta - the difference between two revisions
Merge - determining the outstanding revisions in one branch and applying them to another

* Can run a centralised model, then works quite like SVN

* Or, can run semi-centralised
- Mainline or trunk reflects current mainline
- New features developed in own branch
- Patch Queue Manager (PWM) can enforce this / guard a branch

* Full distributed
- Developers maintain their own branches
- Share / Merge p2p
- Can still use PQM to maintain a test-suite protected mainline branch

* Mixed-mode obviously possible

Advantages
----------
Don't need to give people commit rights to trunk to allow them to contribute (can send their revisions to other developers directly via 'bzr send')

A Bit About Launchpad
---------------------
A web-based tool to support open source software development
Designed to scale

Steps:
* Register launchpad id
* Register SSH keys
* Create branches, push/pull

Get some things for free:
* Server maintenance
* Web view of branch
* Branch notifications (commit mails)
* Simple access control through teams
* Support for code reviews
* Has email interface

OSDC Liveblogging: Openspatial software overview

2008-12-03T15:29:00.001+11:00

------------------------------------
Generally about Open Projects:
- Are there procedures to contribute code?
- i.e. how useful is the project really

------------------------------------
Content Servers
------------------------------------
OSGeo
Mapserver, ,Mapbender, WMS, WFS, WCS

Simple features for sql (SFSQL)
SQL Multi Media Extensions

PostGIS (e.g. used to capture MARS ROVER data, rocket science, important)

GML, SML, GeoRSS, etc
GeoServer is a fully features j2ee geospatial data server

UMN MapServerr, original mover & shaker, high performance

deegree (with two e's) web services is quite comprehensive

GeoNetwork is good for discovering data, is being funded by an Aus govt group

------------------------------------
Applications
------------------------------------

JUMP / OpenJUMP (closed development)
GRASS running since 1982
QGIS
gvSig, open-source funded with 30mil euros public money
uDig, Eclipse RPC based
jGrass

------------------------------------
Thin clients
------------------------------------
OpenLayers is best bet to plug into variety of web-based mapping tools (important)
MapBender

Next year: FOSS4G 2009, Sydney, 500-700 delegates, yearly gathering of the tribes

In response to query regarding gridded data, suggested:
OSSIM: Gridded
GRASS has raster support
GAL+ODR, one of which has raster formats

OSDC liveblogging, Unittest: An under-appreciated gem

2008-12-03T15:11:00.001+11:00

Unittest: An under-appreciated gem
Andrew Bennetts

What makes a good unit test suite?
- readable: intent and implementation is clear
- reliable: pass/fail when it shoud
- usable: easy to run, fast, debuggable

Very quick overview of unittest:
- Subclass unittest.TestCase
- def setUp(self): foo
- def a bunch of methods (will all be test cases)
- single unit test is represented by a TestCase instance
- then run tearDown
- report the result to a TestResult object

TestResult: passed to methods, record success details
TestLoader: turns test methods int TestCase subclasses into test cases
something else: is glue

Why is this good?
- Standard, most other frameworks will interoperate with it, batts included
- Not Python-centric, xUnit implementation, general programmer awareness
- Encourages good structure in tests
- By default, tests are isolated from eachother
- Easy to reuse a test fixture definition between multiple tests (reuse setUp and tearDown)
- Group tests by common needs
- test cases have explicit names, so clearer reporting, can be specific about what failed & where

Unittest is pretty easy to extend.
- extensions have a good chance of being fairly compatible with other test frameworks
- examined addCleanup logic, add cleanup code immediately after, e.g., adding a lock
- (Bazaar extension) multiply_test_suite_by_scenarios is a function that takes a test suite and a list of scenarios

A couple of extra libs worth knowing about
- launchpad.net/testtools: miscellaneous extensions from various projects
- launchpad.net/subunit: runs unit tests in separate processes to support test isolation
- launchpad.net/testresources: is a library to manage the initialisation and lifetime of expensive text fixtures

Some bits are bad:
- no standard tool to load and invoke a test suite
- parts of the API could be better (lots of 3rd party stuff needed)
- set of built-in assertions is a bit small
- documentation in the std lib is a bit weak

- fundamentally quite capable, pretty easy to usefully extend, would be great to get extensions back into std lib.
- launchpad.net/pyunit-friends

OSDC liveblogging - my GAE+ExtJS presentation

2008-12-03T13:17:00.001+11:00

I think it went okay. My presentation covers and introduction to these technologies.

OSDC liveblogging -- Keynote by Chris DeBona

2008-12-03T09:08:00.003+11:00

Chris DeBona oversees open-source license compliance at Google CA.

This presentation talks about how useful OS is to Google. Starts presentation with hardware history of Google, showing Linux ancestry. Current systems are big rack systems, but still running customised linux software with heaps of OS libraries involved.

Google has found about 3.5 billion lines of OS code in unique files, which is more than any of the big players individually have written.

In 2000, asked 10k sourceforge developers about their motivations:
* Intellectual curiosity / stimulation
* skills improvement
* work needs
* non-work functionality
* professional status
* reputation building
* distrust / dislike of proprietary software

Means that OSS developers understand OSS licenses

How does Google use open source?
* Linux kernel
* Apache
* Languages & compilers
* Engineers who run linux (Goobuntu)

Also, when releasing code:
* Android, Gears, apps

Does this to maintain independence from external software company
Allows you as a software company to do something out of the ordinary without showing your hand / without external dependencies

How does Google take part?
--------------------------

8.5 million lines of code released in Android
Release about 1 million lines of code outside of that
Important for engineers to be able to interact with peers in the community, not get out of touch
Provides good code hosting with version control, issue tracking, wiki
Hires some people directly because they are seen as important
Sponsors GSOC -- create new OS developers

Questions
---------

Why didn't Google create their own open source license?
* Don't trust licenses that arent's used much
* Real combination problems when integrating projects
* Not everyone trusts Google, so using well-understood licenses is good for this
* Obvious what Google means by using a license as people are familiar with them
* Developers don't need to understand new legal terms, familiar with known licenses
* Uses Apache a lot because it has specific things to say about patents (personal comment -- helps avoid patent issues -- not exactly sure how but seems to provide some end-user protection against patent infringement through using the software)

Now running out of battery, so posting now...

OSDC08 Day One

2008-12-03T08:48:00.002+11:00

Yesterday (Tuesday 2 Dec, Sydney AUS) was day one of the OSDC 08 conference. It began with a Google Hackathon, where a few Googlers presented some of their latest technologies. These includes both open source projects which they sponsor (or at least contribute developer time) and closed-source but free-access systems. These systems were:
* OpenSocial
* Something about an open map layer system
* AppEngine

This was pretty neat, and I successfully copied-and-pasted my way to a functioning OpenSocial application which would allow me to give my friends virtual acorns in a variety of social network containers. Google make available a number of things to make this easier, including comprehensive examples, a number of free hosting solutions and a sandbox environment for Orkut to deploy test applications.

I haven't been active in this area of development before, but a great deal of the complexity of managing such an application has been stripped away to allow any potential developer to concentrate on application functionality instead of systems maintenance.

Following this, I visited the Google offices as a part of the Sydney Python User's Group meeting. At this meetup I presented on the topic of AI in Python. At this stage, I prefer not to make my slides web-available but would be happy to supply them via email to anyone at the presentation or who would like to engage on the topic offline. Since AI is such a huge area, with so many potential audiences, there is a lot of ground for misinterpretation of a slide or bullet point if taken out of context. I had great fun however, and greatly enjoyed catching up with everyone at the pub afterwards.

Props to Christoph, Mark, Alan, Richard, Mark, Matt, Alex, Joel and everyone else that I met whose names I can't right now recall. Thanks for making my stay a nice one!

-Tennessee

IT is not like upgrading a car, like designing a house

2008-11-27T09:54:00.001+11:00

I would contend that IT problems tend to be organisational problems rather than technology problems. Upgrading your IT is not like upgrading your car, where you just get better performance from your wheels, accelerator and brake. Rather, upgrading your IT is like designing a house. Yes, new houses are often built from better components, but you still need good design that meets your particular needs. To some extent you can buy off the plan, but the result is never as good as working closely with the architect.

Book Meme

2008-11-13T20:57:00.002+11:00

INSTRUCTIONS:

* Grab the nearest book.
* Open it to page 56.
* Find the fifth sentence.
* Post the text of the sentence in your journal along with these instructions.
* Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

"Not only is the flowering of global science and technology not an exclusively Western-led phenomenon, there were major global advances in the world that involved extensive international encounters away from Europe".

This week finds me reading "Identity and Violence" by Amartya Sen (subtitled "The Illusion of Destiny"). Last week would have found me reading Terry Pratchett, which probably would have made for better blogvertainment...

Cheers,
-T

Mashup idea: library catalague with book reviews

2008-11-12T09:46:00.001+11:00

I'd like to see a really good book review site that could talk to my local library catalogue system, so that I could check out what highly-reviewed books my library currently has available for loan, and perhaps even use it at the basis for purchasing requests. Just another way that Things Could be Better! :)

-T

Pythonista, pythoneer... pythonite

2008-11-06T10:10:00.002+11:00

Another noun for describing a python afficionado. See http://www.wordinfo.info/words/index/info/view_unit/1096 .

As in socialite, canaanite.

Also related to the rare substance Pythonium, the pure essence of Python in mineral form. Pythonite is a substance created from Pythonium. Consuming Pythonite (or refined Pythonium) can lead to greatly enhanced productivity. Hence the somewhat ambiguous term Pythonite may refer either to a Python afficionado or the mineral substance formed when Pythonium is present in the natural environment.

Can react energetically with Perlite despite its proximity on the programmatic table of elements.

-T

More on soft exceptions

2008-11-05T11:58:00.003+11:00

I had a real-world problem just now which I chose to tackle using soft exceptions. I have a series of rules which transform a data object. Depending on the state of the data object, not all the rules are applicable. Also, depending on the rule, it may or may not be critical to have the rule applied to the data object.

While pursuing another task, I caused an exception inside a method which applied one of these rules. Rather than simply track back the exception and add more handling to the code, I decided to *also* wrap this rule in a soft exception handler. In operations, it would be far better to simply not apply the rule than to cease processing and crash, yet the rule logic is written such that any error causes a program fault.

Ideally, I would implement the soft exception handling using a logger level which could be applied application-wide, but that is a bigger task than I wanted to tackle in 10 minutes!

So, I wrote:


exceptionMode = "Soft"

try:
  originalState = object.state
  // Apply rule logic

except:
  object.state = originalState

  if exceptionMode == "Soft":
    print "NEW WARNING -- generated in MyClass.methodName
    print "Methodname failed. Original state is being used."
    traceback.print_exc()
    print "END WARNING"

  else:
    raise

return successFlag

This worked beautifully! I can check the code in with soft exceptions turned on, so that in operations any bugs here will only result in a rule failure, not a complete process failure. However, I still have access to the full stack trace through logging and can also set the exception mode to "Hard" (or anything that's not "Soft") and have 'brittle' behaviour for testing.

Score 1 for soft exceptions!

Clearly, it would be ideal to log instead of print. It would be good to have have the following logging controls:
* Hard or soft exception handling
* A logging level which would actually bring up user warning dialogs
* A variety of logging detail levels

Cheers,
-T

Meme: Top Five Software Possibilities

2008-11-03T13:34:00.002+11:00

Here's a quick meme... List the top five things that (you believe) could be feasibly achieved with a software R&D investment in the order of $30 million. Please list a time frame for any particularly big ideas. They could be your top five moneymakers, top five in terms of benefit for society, or anything -- it's YOUR top five :)

1. Flawless text-to-speech rendering. Could be done in four years, will take 15 otherwise.
2. Driverless cars. Could be done in six years, will take 40 otherwise.
3. Complete automation of the tax system. Maybe six years?
4. Creation of fully anonymous, publicly available medical research datasets.
5. Create an artificial mind. Could be done in ten years, but may never be done.

What are your top five?

-Tennessee

My project: GAE-JSON

2008-11-02T19:02:00.003+11:00

URL: http://code.google.com/p/gae-json/

The project is really not ready for public consumption, but I have noticed another project called gae-json-rest which is turning up on Google search. So that others can find my project by searching for it, I just needed to create a couple of links to it so that it will be available. Not, of course, that I'm being competitive -- but if I don't put some links up then there will be a lot of confused visitors to the gae-json-rest project!

If anyone feels very enthusiastic about the project, there *are* a couple of functional examples which anyone could install on their system, but they will need some guidance first. By the time OSDC08 comes around, however, I intend to have turned this around into something which is really genuinely user-friendly. I should add to the list of my goals:
* Create a hosted demo
* Put together a wizard / walkthrough
* Create some documentation

However, goals at the moment will still be those relating to functionality...

-Tennessee

This close to a datastore --> json server on google apps engine

2008-10-31T16:45:00.002+11:00

Phew ... a good afternoon's hacking.

I am *this close* to completing a very basic prototype of a json server for google apps engine. The idea is that you visit a URL which contains the query you want to evaluate (hence my previous post) and get back a JSON encoding of the data. Obviously, that's for requests only.

However, what it *means* is that you can then write an AJAX client which can interface directly to a GAE data store. And *that* is a good thing.

Sure, it's about as functional as a car with no steering wheel, but in principle it works. I have achieved the following goals:
* Store some data in the data store
* Retrieve it with a GQL query
* Write a web server which decodes a query from the URL in hex
* Use that to retrieve said data from the query store
* Render the data to a web page in HTML
* Render the a json string to the web page (or json-looking, anyway)

The following goals remain:
* Write a version which emits the json string with appropriate mime type
* Write a javascript client which retrieves the data and renders it in a web page
* Put up some vaguely comprehensible web pages explaining the thing and demonstrating its use
* Make the store/retrieval more general by:
-- Not relying on a specific database structure
-- Going beyond string-only data
-- Do something about INSERT queries

I'm having some fun with this. Anyone who is interested, let me know and I can keep you posted of progress. I think it should be really useful.

Efficient encoding of URL arguments

2008-10-31T14:44:00.002+11:00

I would like to have URLs which look like:
"http://web.host/queryhandler/Tnidf3445ssx34" or something similar. That last part should be some reasonably space-efficient encoding of something like "SELECT * from FOO where CONDITION"

I have found I can get part-way using Python's encode and decode functions, e.g.
encoded = "SELECT * from FOO where CONDITION".encode("hex")
decoded = encoded.decode("hex")

However, the resulting values are still a bit long. This method quite nearly removes nonprintable characters, spaces etc from the URL. However, I have this nagging feeling that I could be doing some compression along the way to make the encoded part shorter. For example, HEX uses only part of the alphanumeric space. I'd like to use the digits 0-9 and letters a-z and A-Z ideally.

Let's just make this clear -- for no very good reason that I can justify, I want to encode a query as the shortest possible string using printable, alphanumeric characters (and no spaces) and be able to simply decode the string also.

"The Internet" has told me about md5 hashing, but I need a two-way function here. The help on string.encode in Python simply says I can specify the name of any encoder, but I can't find a list of the default encoders available.

Any ideas, internet?

Cheers and thanks in advance,
-T

OSDC 2008 pre-conference activities

2008-10-30T13:54:00.002+11:00

I notice that the OSDC conference days have dropped from the 1st - 5th
to the 2nd to the 5th as the conference schedule has been firming up.
I find myself now with a free day on the 1st. I wonder if there are
any Pythonistas in Sydney who would care to catch up on that day for a
discussion group, BoF sessions or anything else? Looking at the
program, it seems to have fewer of such events than past conferences.
There are some available slots for these sessions, but if anyone wants
to get together for some more in-depth discussions, I'd be up for it.

I'd be very interesting in informal sessions on the following topics:
* Open source in government
* Weather forecasting
* AJAX development
* Artificial intelligence
* Advanced Python skills
* Open Source & Project Management

I have also posted this message to various Australian Python mailing lists. So far there are two takers for some activities on the Monday. If interested parties can email me, I will see if there are enough takers to organise some small-group discussions on topics of interest. So far the takers are 1 for Open Source in Government and another for AI.

Regards,
-Tennessee Leeuwenburg

Automatic text generation is live!

2008-10-29T14:14:00.003+11:00

Hi all,

The system which I have been working on for three years has just gone live. My component of this system generates text forecasts automatically, based on gridded forecast data which is initialised from model conditions and then manipulated by weather forecasters. Any particular text forecast may also be edited by the forecasters after generation, but the bulk of the forecasts are published without forecaster intervention in the text.

Examples include the Melbourne forecast,
http://www.bom.gov.au/products/IDV10450.shtml

A number of other Victorian town forecasts, Victorian district forecasts and an expanded set of Precis and Town forecasts are also produced. Also, a larger number of forecast days are included for many of these locations.

This page (http://www.bom.gov.au/nextgen/) describes some of the changes to service which come with the new forecast system.

Cheers,
-Tennessee

Code, wiki style

2008-09-26T15:10:00.003+10:00

I've got this use case for editing code through a web interface, like a wiki. I thought I'd find out if anyone else has thought about this topic, and didn't see anything jump out at me when searching.

The situation is that maybe there's an open-source or perhaps just shared-development project in a workplace, hosted on the cloud. Rather than each person having an IDE set up for them or accessing a CVS repository, they just have a web interface.

Maybe the system requires logins to track changes, or in the first instance maybe it just allows unrestricted and un-tracked editing.

Does anyone know of such a system? Ideally, the server code would be written in Python (I'm trying to set up a Python-only technology stack for reasons of minimalism) but I would be interested in non-Python systems. I don't mind if the interface is fairly primitive to start with, but it would be neat if it did include some additional features such as syntax highlighting.

What do people think?

Cheers,
-T

search through code, search through comments

2008-09-16T13:41:00.002+10:00

Dear lazyweb,

I would love it if I could search through my code using my IDE, with the following options:
* Search all text in files
* Search only comments
* Search only non-comment code

My current favourite IDE is Eclipse/PyDev, but I'm not sure where to lodge queries easily, so here's my message in a bottle for the relevant developers. God speed, little bottle!

Cheers,
-Tennessee

OSDC 2008 GAE + ExtJS code samples

2008-09-15T15:31:00.002+10:00

This page contains links to demonstration applications developed for OSDC 2008, including:

* JSON server written in Python integrated with GAE and the Document Store
* Basic ExtJS form submission to Google Document Store

For GAE-JSON SVN server, visit: http://code.google.com/p/gae-json/

Soft exceptions

2008-08-27T13:35:00.003+10:00

Something that I would like to consider for the future is an idea I call soft exceptions. They are probably industry standard already and I'm just the last to know, but here's the idea.

Many times I want to execute a block of code which does something, but not something important. If I'm writing some important piece of software, I don't want problems in unimportant code areas bubbling up to cause big problems in areas that *are* important. This can be managed by writing great code, having an entire bureacracy of test scripts and good defensive coding habits. Or you can just wrap the whole lot in a try statement and best practises be be blowed.

I've got this idea that there can be a middle ground -- a way of being able to protect your important code when it's live and being relied upon (other than by just not having errors happen) while not putting in kludge after kludge to keep the motor running.

The idea is, basically, that exception handling would be linked to the logging system, and that whether an exception was fatal or not would depend on the logging/debug level set.

For example, I might have a code loop which is something like the following:


while(True):
     doSomethingReallyImportant()

     try:
          doSomethingNotVeryImportant()
     except KnownFailureMode:
          print "Sorry, there is no wheat in the hopper"
     except:
          log("Something not very important broke unexpectedly")

          if logLevel = "soft":
               log('soft exceptions, see traceback')
               log(traceback.format_exc())
          else:
               raise

This code would
a) Protect the important code in the loop
b) Respond to known failure modes
c) Log everything
d) Re-raise unexcpected exceptions if in debug mode
e) Suppress unimportant things if in operational/critical mode

I could even imagine "fallback mode' operations which could be user-enabled which would enable hard exception handling....

Just some thoughts....

Melbourne-based Javascript experts

2008-08-15T11:51:00.003+10:00

Hi lazyweb,

I was wondering if there are any Melbourne-based javascript experts who would be able to help me with a bit of learning. I could really benefit from being able to pick someone's brain for a few hours, with problems in front of me etc. I don't want a training course, just a kind of expert tutor for some assistance. I'd be happy to pay a fair hourly rate for the time spent of course. Google a.k.a. the Internets didn't really have any obvious promising leads, so I'm hoping this might find its way into the inbox of a real person who could help me out. God speed, message in a bottle!

Email me at tleeuwenburg@gmail.com.

Thanks,
-Tennessee

Web application design philosophy - HELP!?

2008-05-30T15:41:00.002+10:00

Full disclosure: I started developing web apps like two days ago. This post is written as a n00b, not as a source of authority.

Not to be intimidated, I got out my slate and chalk, and began doing screenshot designs for a rich internet application (RIA) I want to write. I found myself some whizzbang technology, in this case Google Apps Engine and the Extjs javascript widget framework. They're both pretty easy to work with (so long as you don't need to do any debugging) and I managed to prototype myself some pages to do Neat Stuff.

I'm now faced with a difficult path to actually learning how to write a web application "properly". With a standalone GUI app, I'm pretty much fine with application design, breaking things up into an MVC model and then doing an object-oriented design of each bit. With a web application, it's not so clear how I should do things.

If I were designing something in static HTML, all the logic goes on the web server. The user clicks "load data" and the web server goes and produces a static HTML page showing the data loaded. The application brain is in the server, and the purpose of the HTML page is to render the user interface to the client.

It appears possible to basically do the complete opposite with javascript, and put basically all the processing logic into the javascript, treating the web server as just something to go fetch resources from.

So should the logic go?

I'm worried that if I pursue a heterogenous design, whereby the application logic is split between client and server, that I will then have an unsatisfactory level of coupling between the two. Should the database be the 'model', the web server be the 'controller' and the client be the 'view'?

Presumably, if I put heaps of logic into the javascript, then network overhead will be high but latency will be good? Whereas perhaps if I do it the other way, network overhead will be low but many more requests will be necessary, thus slowing application responsiveness?

Where do XML web services and SOAP requests come in? "Should" I be using these as an API, or should I just request data and perform my processing in javascript?

How do I write a web application such that it is reasonably 'pluggable' -- that is such that I could alter my choice of server, database or javascript extension library without killing myself?

Should I use something like Django's templating to control the GUI or should I put all the GUI design into my javascript? Should there be a javascript object representing the whole application which then loads the navbar, display and other widgets?

Should I do form processing on the web server, or in the javascript?

ARGHHH!

I think, maybe, the index page should be a trad "web page" kind of page, with links to the RIA pages.

-Tennessee

OSDC 2008

2008-05-28T19:54:00.002+10:00

From 1 to 5 Dec, in Sydney, Australia, the Open Source Developer's Conference 2008 will be held (http://osdc.com.au/2008). I will, in all probability, be attending this. I would love to get in touch with any other Pythoneers who may be going. It would be fantastic to organise some group sessions and do some networking. Also, if there's anyone else who works for the Aus govt it would be good to hear about what kind of open source applications are getting a guernsey in your depts.

I thought it would be neat if we could find some leading Aussie python people and rope them into giving a panel talk on some topic, leading a discussion or some such.

Advanced Document Notes

2008-05-20T16:52:00.002+10:00

I just started a Google App Engine trial and my goal is to prototype an application I've always wanted. If it works, I will even use it. It's really simple to explain -- I want to take notes against documents. Then, I want fulltext search through everything.

There are files. Some are public some are private. Let's assume they are all PDF articles that I want to write about.

There is a common document store and a private document store. Documents can be shared, and you can read eachother's notes! Yay! But some are private, because sharing them would be bad for some reason. So you have access to the global store (which is like a wiki) and the private store.

Okay, so you log in, and can upload a file. Yay! Then you take notes against it, which is stored in a notes file and related to the original file.

Then, you can have a display where the PDF is viewed on the left and the notes are viewed on the right.

So, you could share something like a PDF, text file, web page or even binary media file, and share comments on it. I envisage it as being great for the following uses:

1) An academic who wants to create a personal store of research notes
2) An academic who wants to collaborate on creating research notes, such as with fellow researchers or students
3) Anyone who wants to share comments about files.
4) A workplace where a group wants to share around a file for comment

I just hope I can find time to actually do this. I don't think it will be particularly hard, but neither do I have a lot of free time.

Anyway, cool! GAE!

I'm trying not to think of any additional functionality at this stage, but ideas keep popping into my head anyway.

-Tennessee

Requirement: Taking notes against a document

2008-03-23T15:13:00.003+11:00

Here's a tool that I want. It's simple, and clearly useful. I study, as do many people, and I think the use case goes further. I have documents, which I don't need to edit -- e.g. articles and papers. I need to take notes against these documents. Further, I would like to be able to search within the body of documents and notes. I want to be able to create an articles hom directory (~articles). I then want to be able to have my own subdirectory structure (~articles/subject). I then want to be able to create notes files that are associated with each article. Then, I want a search tool (like Beagle or Google Desktop) which will constrain itself to ~articles, and know about the association between notes files and articles. I want full-text search, with a link on the results page to both the article and the notes. That would be great!

-Tennessee

From my python interpreter

2008-02-20T12:19:00.001+11:00

>>> a = 1,2
>>> a
(1, 2)
>>> a == 1,2
(False, 2)

Article Notification: The Impact of Component Modularity on Design Evolution: Evidence from the Software Industry

2008-01-25T11:42:00.000+11:00

This paper interested me and I thought it might interest others.

Title: The Impact of Component Modularity on Design Evolution: Evidence from the Software Industry
Authors: Alan MacCormack, John Rusnak, and Carliss Y. Baldwin
URL: http://hbswk.hbs.edu/item/5831.html

Executive Summary (from the site):

What factors should influence the design of a complex system? And what is the impact of choices on both product and organizational performance? These issues are of particular importance in the field of software given how software is developed: Rarely do software projects start from scratch. The authors analyzed the evolution of a commercial software product from first release to its current design, looking specifically at 6 major versions released at varying periods over a 15-year period. These results have important implications for managers, highlighting the impact of design decisions made today on both the evolution and the maintainability of a design in subsequent years. Key concepts include:

* Data show strong support for the existence of a relationship between component modularity and design evolution.
* Tightly coupled components have a higher probability of survival as a design evolves compared with loosely coupled components.
* Tightly coupled components are also harder to augment, in that the mix of new components added in each version is significantly more modular than the legacy design.

comapping.com

2008-01-22T16:06:00.000+11:00

I just found the most astoundingly useful web tool. There's a free 30-day trial and I highly, highly recommend that everyone take it for a spin. It is, essentially, an intuitive structured-note taking tool. It appears to create notes in a tree-based structure, with node types which correspond to projects, goals, tasks, notes, meeting agendas etc depending on your chosen template.

http://comapping.com/

Cheers,
-T

Developing for userspace (linux-focused)

2008-01-03T09:22:00.000+11:00

On great Python utility is virtual Python. It is a script which will take your system Python install and leverage that to create a Python binary in your home directory. The advantage is that you can then install all kinds of third-party libraries without (a) root permissions or (b) messing up your primary installation.

It can be used to create several different "homes" for Python, which could let you experiment with library version, install procedures or a variety of other things.

More people should develop for userspace.

I was recently trying to install C#, something which I would love to learn. On linux this means mono. On a standard operating environment without root priveleges, this means a *lot* of dependencies (due to old versions of Stuff) and a lot of compiling.

It would be great if RPM (and apt, or pick-your-favourite-installer) could handle the idea of a userspace package repository. Maybe they can and I don't know about it, but I've not seen any information on the topic. Things that get installed should have a userspace option. If I want to grab the latest gnome, mono and C# libraries into user-space without root priveleges, I should be able to. I should be crippled just because I'm not root. It doesn't seem beyond the capacity of linux to allow users to do this, since it already supports the userspace paradigm very well.

-Tennessee

A project I would start if I could

2007-12-20T12:37:00.000+11:00

LaTeX 2 ODT converter. Probably couldn't accept arbitrary LaTeX, but should at least be able to do headings, subheadings, lists and formulae/diagrams via images.

This could easily be run as GSoC or GHOP.

Interview -- "Why They Do Python"

2007-12-18T09:59:00.000+11:00

I was recently asked to participate in an interview of a panel of Python experts by Odin Jobs, a recruitment site. Happy to help raise the profile of Python, I accepted. The interview, which includes ten other Pythoneers, is now available at http://www.odinjobs.com/blogs/careers/entry/python_experts_why_they_do.

My entry for James Tauber's Competition

2007-11-09T10:24:00.000+11:00

I coded up an entry for http://jtauber.com/programming_competition/. Averaged across the categories, I'm coming about fourth/fifth. I may or may not try to refine my algorithm further, but it was fun taking part. Here's my unpolished code for the world to throw rocks at.

The algorithm is supposed to work on a reverse logic, considering which word affects the least number of verses and learning it last. It's too slow to handle the category four input file (I reckon it could take a week). I wrote a modified version which is much faster but does not perform as well.

The major time code is recalculating the word frequencies in each step. If this could be sped up, the algorithm would be tractable for larger inputs. The modified version runs in just a few minutes.


import sys


def remove_partial_verses(known_words, verseDict):
    known_set = set(known_words)
    for verseID, wordSet in verseDict.items():
        if known_set.intersection(wordSet):
            del verseDict[verseID]

def countWord(word, verseDict):
    '''
    This method counts the number of verses which contain the word
    '''
    count = 0
    
    for verse in verseDict:
        # count the number of verses
        wordSet = verseDict[verse]
        if word in wordSet: count += 1        
    
    return count

if __name__ == "__main__":
    importFile = sys.argv[1]

    inputFile = file(importFile, 'r')
    
    wordDict = {}
    verseDict = {}
    
    for line in inputFile:
        verse, word = line.split()
            
        if verse in verseDict:
            wordSet = verseDict[verse]
            wordSet.add(word)
            
            verseDict[verse] = wordSet
            
        else:
            verseDict[verse] = set([word])
            
        if word in wordDict:
            verseSet = wordDict[word]
            verseSet.add(verse)
        
        else:
            wordDict[word] = set([verse])

    totalWordSet = set(wordDict.keys())            
    known_words = []

    learn_order = []
    
    while(len(totalWordSet) > 0):
#        print "Length: ", len(totalWordSet)
                
        pairs = [(countWord(word, verseDict), word) for word in totalWordSet]    
        pairs.sort()
        
        theWord = pairs[0][1]
        learn_order.append(theWord)
        known_words.append(theWord)
        totalWordSet.discard(theWord)


        remove_partial_verses(known_words, verseDict)

    for word in learn_order[::-1]:
        print "learn ", word

NVIDIA drivers when upgrading from Feisty to Gutsy

2007-10-31T12:16:00.000+11:00

This mysteriously stuffed up on my system. The key, it turned out, is that there is a new package name for the nvidia drivers. The old package is not replaced during the upgrade, nor does it totally fail. It just doesn't work right.

The key is to turn off the proprietary driver, then turn it on again. This will trigger the de-installation of the old proprietary driver and the installation of the new package.

ExtJs and Python

2007-10-29T16:54:00.000+11:00

It appears that extjs (http://extjs.com) is the greatest thing since sliced bread.

Since Python is also the greatest thing since sliced bread, they must be the same thing.

My goal is to start using a Python backend to support an extjs frontend application of some kind. I'd love to hear from anyone else who has used extjs, especially if they have tried to integrate it into a Python web framework.

I'll keep readers posted as I continue development of this application. It is likely to be very slow development, however, as I need to juggle a number of other responsibilities.

Cheers,
-T

First Post -- Newsgroup/Blog integration

2007-10-29T16:27:00.001+11:00

As Editor-In-Chief of The Python Papers, it is not always appropriate to blog everything I would like to the official blog. I have decided that I need another blog. Everything that goes in pythonpapers.blogspot.com is said with my Editor-In-Chief hat on. Everything said on this blog is said with my own hat on.

I also ran melbournephilosopher.blogspot.com, but am no longer commenting on philosophical issues.

One thing which I'd really like is the ability to link my blog posts with a newsgroup. It would be rather awesome if everything I blogged here would get posted to comp.lang.python, and all replies to that message got left here as blog comments. That would be tres cool.

Thanks,
-Tennessee