Tuesday, June 30, 2009

Call for interest: should I run a python AI competition?

This is just a shoutout to see whether there would be any interest in my running a Python AI competition? There are a few Pythoneers who are into AI that I know of, and it occurred to me that one thing which could be done to serve the community would be to offer a competition.

At this stage, I'm thinking either of challenges which relate to basic AI algorithms, or perhaps building a chatbot/twitterbot. The first challenge should probably have a low barrier to entry, so perhaps I will put together a multi-stage or multi-challenge competition so that people can choose their own level of competition.

Leave a comment with (+0) if you think it would be neat, (+1) if you would take part, (-0) if you're not really in favour, or (-1) if you would sabotage the competition :)


Thursday, June 18, 2009

Python, running in a browser


I have a dream: one day, I will be able to not know Javascript. Thanks Mark!


Wednesday, June 17, 2009

Driving unit test takeup with code coverage

For anyone who has not fully gotten organised with unit testing and code coverage, this is for you! :) My project involved a large inherited codebase which has good black-box testing, but little unit testing and no coverage metrics. Tackling unit testing was always something impossible for me -- how do you take 96, 000 lines of Python and 1, 000, 000 lines of C, and build unit tests? (nb the lines-of-C count seems high, but it's what 'wc' said.)

The general advice is not to try -- but to get traction by just writing one unit test for the next bit of new code you write. Then, the next bit, etc. Eventually, you will have figured out unit testing and will then make an appropriate judgment on what to do about the body on untested code.

I have typically found this to be quite a large hill to climb. I work only on a subset of the code, which practically requires me to invoke most of the code just to get to my part. Most of my methods require such a lot of setup that it seemed quite infeasible to tackle unit testing without doing some more thinking about how to do this in a sane way. Setting up and tearing down my application was just not going to be feasible if I were going to put a lot of unit tests in place -- I reckon the setup would have cost between 1 and 7 minutes per test!

This got relegated to the too-hard basket, and set aside until now. Here's how I found a way to get traction.

What turns out to be pretty straightforward, is integrating coverage testing. You more-or-less just switch it on, and it will record to a file across multiple program invocations. This can be used to count coverage across test scripts, or user testing, or development mode use, or indeed in operations.

I ran this through about half my black-box tests, and found I was sitting at around 62% code coverage. That's not too shabby, I reckon! I know for a fact that there are quite large chunks of code which contain functionality which is not a part of any operational code path, but is preserved for possible future needs. I estimate 25% of our code would fall into that category, lifting the coverage to 87% for the sub-area of code which I work on. Now I've got numbers that look like a challenge, rather than an insoluble problem!

I think that's the key... make lifting code metrics an achievable challenge, then it will seem more attractive. It's probably important not to target a particular number. I know '100% or bust' may be what some advocates would put forward, but for anyone new to the concept, I personally feel that simply measuring the current coverage, then understanding where that number comes from and what it means, it the more important achievement.

What is clear is that I'm not going to easily lift my coverage metrics beyond a certain point simply through tactical black-box testing. I'm going to have to write tests which go through the code paths which aren't part of the operational configuration. I'm going to have to write tests with very specific setup conditions in order to get to lines of code which are designed to handle very specific conditions. All of a sudden, I've got achievable and well-defined goals for unit testing.

I call that a win!


Anyone been to an Australasian Joint Conference on Artificial Intelligence

Dear lazyweb,

Has anyone been to one of these conferences in the past? Did you find it valuable as an attendee? Please feel free to email me directly at tleeuwenburg@gmail.com if you would prefer not to leave your comments publicly.


Thursday, June 11, 2009

The Pinball Principle

I read a fair bit about different development methods. Agile is pretty popular amongst developer advocates, but I would argue almost everyone in charge of development *actually* just runs with a grab-bag of "what works for them" taken from Waterfall, Agile, what their peers think, what happened on the last project etc. They'd be mad not to -- can you imagine running development based on what doesn't work for you?

Some people talk about bottom-up vs top-down, inversion of control, directed vs undirected, the importance of use cases, test-driven development etc. Usually, the argument is that if you tick whatever list is being advocated, better software will get written in an efficient amount of time and deliver what was ordered to the greatest extent. But, somehow, it never seems to work out that way.

There is a huge complicating factor, which is that most development activity is done due to an obligation owed by an employee to an employer, client or other stakeholder. The developer ends up responsible for not just managing their own development practises, but managing their clients through the process also. If there are just a few stakeholders, then that's easier (unless they're very difficult stakeholders). Issues come up when there are multiple stakeholders, and when there is conflict between those stakeholders. The best possible course of action ceases to be "build and deliver approximately X" and becomes "make all these people happy to the extent possible".

How does making people happy fit in with SLDC practises? Where do you stand firm, and where are you flexible? When do you tell your manager "we shouldn't do that" and when do you just fit in? How do you sell a use-case Agile practise to a control-oriented manager? What are the ethics of accepting your manager's control-oriented activities if you think things could be done better another way? When do you take a decision on your own shoulders? How do you actually adopt new design practises? How does it fit into the development team and the organisational culture?

Most people in an organisation run according to what I call the 'pinball principle', which is to say they get batted around by their key stakeholders, bouncing off obstacles, temporary victories, other groups etc like the objects in a pinball game. To some extent you can control your own fate and direction, but there are a *lot* of things which you can't control.

In a large development team, a kind of 'surface' will form around the team, representing that team's culture, which can protect those inside of it. However, some people either don't have a protective surface, or they are right on the edge of it. In these cases, you can't simply adopt a practise and hope it will result in better outcomes, you need to use personal insight to negotiate the situation you find yourself in, and no amount of 'best practise' will prevent your key stakeholders looking you in the eye and demanding action, their way.

The question is whether there is anything we can learn from the pinball principle. If our environment is so complex and demanding, is there anything about that which should inform our behaviour? Should we acquiesce to all incoming requests, devolving responsibility, and concentrating instead on doing the best job we can? Or will something like the Agile methodology allow us to write great componentry which will then turn out to be just the thing for the next request? Should we attempt to educate stakeholders in appropriate negotiation processes, or should we instead rely on meetings and face2faces to take care of things. What is the cost of attempting to enforce process?

If project direction is likely to be set by pinball mechanics, how can we best write software knowing that will happen?

I don't know the answers -- not all of them anyway. But here is *my* list of things which I think will help keep the development *direction* on-track, and best serve long-term development success. I have just come up with them now, but they are based on ideas which I have been refining for some time now. I'd love feedback on the list and this post!

1. Get all of the stakeholders into the same room at least once a month. Make them review their priorities in front of everyone else, and get agreement on the priorities for the development team.
2. Maintain a whiteboard or wiki with the top four development priorities. Focus prioritisation on the top four issues.
3. Email out a bulletin every fortnight without fail to all stakeholders and users who are interested, including the current priority list and progress achieved.
4. Hold 'air your grievances' sessions regularly (say twice a year), especially if there is a layer of bureaucracy between you and your end users. Don't minute them.
5. Identify what makes your stakeholders actually happy (i.e. less user complaint, positive feedback from other departments etc) rather what what they say they want (more progress for less money). You will only make other people happy by reducing the pressure they get from the pinball board. That's only sometimes the same as what they're asking you to do
6. Pursue a two-layer development methodology. Make the bottom layer whatever you like (Agile, TDD, Waterfall), and make sure the developers understand it. Keep on top of new ideas here. Make the top layer As Simple As Humanly Possible. This is your 'interface' process, and keeping it simple is the only way that others will follow it. If you can't explain it at the beginning of every meeting in two slides, then it's too complicated.
7. Document Everything.
8. Publishing = profile = perception. Make sure you nail the following items: project web page, prototype web page, email bulletins, conference/journal publications (if relevant), regular meetings/presentations