Monday, May 1, 2017

PyCon Australia 2017, SEO and things that should be less work

So, I have an unexpected problem. I'm helping to run a major Australian conference, and we can't seem to get found by Google. This year.

Last year's conference is no problem. There are many, many links to that. But somehow, this years' site,, is just not cool enough. Maybe PageRank changed. Maybe Google and angry at Twitter and not regarding tweets as important contributors to page significance. Maybe they just haven't gotten round to adding each mention of @pyconau on twitter into their page rankings for our site. Who knows?

I got myself added to something called the "Google Search Console". Do you know how many links there were? Guess. My guess is that the real number is less than you will guess.

I'll wait.

There are eight links. On the whole internet. Tim Berners-Lee would be disappointed. Somehow, our new sharing based technologies have left simple "search" behind, doing away with hypertext linking between a vast network of connected documents, and replacing it with another concept entirely.

I have no idea how this will affect our conference. Perhaps we're leaving Google behind as a concept, and search is losing relevance compared to the significance of "going viral" and tapping into realtime mindshare. Perhaps it'll be our best year yet.

But I like to believe in something else. That all that hype is just "surface layer", and that old school linking to things still has a role to play. That the internet isn't a "feed" but an organism, and if we help to do the gardening, it will grow and flourish.

So, if you have a minute, please consider making a link, from your site, to ours.


Friday, January 6, 2017

Inexplicably earning 30c

I'm up late tonight. No particular reason, I just get a bit of insomnia sometimes. It seems best not to worry about it, but just to ride it out and do whatever seems to keep me calm. I wrote a few blog posts late last year, spurred by having looked at some of my blog view stats and looking at my adsense total earning. At that time, I had $14.68 of credit I could call in.

I wrote a couple of posts. For me they got a high number of views -- about 5600 and 5800 accordingly. I even got some "plus ones", and better yet, some comments. That led to feeling moderately positive about my efforts, and I liked it.

Interestingly, my available credit has since gone down to $14.23. I assume this is due to fluctuations in the AUD vs the USD rather than anything else, but I suppose it could be that credit earned a long time ago is expiring.

I had a look at the big bucks being pulled in via advertising. I earned 1c and 2c respectively for those posts. I'm a bit confused though. Google put my pageviews as 25 and 43 for those two posts when I look in AdSense, but I'm seeing numbers in the thousands when I look in the Blogger interface. I would assume that between 25 and 43 is possibly the number of real humans who took the time to read some decent chunk of the articles and got something from the experience. Frankly, it's nice to think I'm providing some kind of value to a few Real Humans somewhere or another, I'm not fussed about the numbers.

That's to say, I'm not emotionally invested in getting larger numbers. The curious part of me is seriously bugged about the inconsistency in the statistics, and the relationships between "Blogger Views", "AdSense Views", "Clicks", and earnings.

I'm wondering whether the Blogger View count is being deliberately stupid, so as to factor in the vas army of crawlers, bots, search engines and random traffic. However, that seems less likely when I consider that my articles do get high or low views in a way that I can see relates to my own impression of how interesting they are. Either my better posts are getting crawled in more places, or else they actually are people clicking on things. I suspect if I spent enough time reading through the internet I'd find some kind of explanation. But the point of this blog is to make the actual thinking process along the way more interesting and more shared, rather than to engage in a solo effort to game adwords. Maybe I'll go looking for more facts later.

--- quick update ---

Okay, based on a quick search, it seems likely Blogger is including every damn hit to the blog, whereas Adsense is doing a much better job of getting at actual human with web browsers looking at things (excepting those with JS turned off or with adblock which I suspect/hope is most of my readership anyway). I mostly think it's a shame that Blogger has such a misleading view count, if less-technical people are getting a false impression of how popular their blog actually is.

--- back to original post ---

The most interesting thing to me was the day I earned 30 cents. This is, by far and away, my best earning day in recent times. It didn't occur on a 'posting' day, or even in its vicinity. As best I can guess, a single random visitor chose that particular time to actually click an advertisement. For reasons which are good and proper around privacy, or perhaps just implementation cost, there's no feature in the system which actually lets me discover who it was. If it was you, and you remember clicking an adword advertisment on Wednesday 19th of October 2016 (Australian Eastern Daylight Savings Time), leave a comment!

One thing that was interesting, is that click sent the value of my page views and impressions through the roof. On reflection, that stands to reason. Blogs with "clicky" users also have more potential energy stored in their views and impressions, also increasing their potential value.

I worry about whether this is about to commit a minor infraction against the terms of service, but hopefully if you've read this far you don't tell anyone. If you can go and deliberately click on one of the adwords, I'll be able to get some more data about the statistical relationships involved.

In the meantime, I'll rest happy that most of my readerbase is too smart to even notice internet based advertising, let alone click on it.

Subsequent thought ... I suspect I could choose ads better than an algorithm for my readers. If I could just wire the advertisement permanently to ThinkGeek, I would probably generate more clickthroughs. Back before I had adblocker installed and set to 'kill everything' mode, the ads I saw typically had little relationship with my actual interests. I learned more about the kind of companies choosing to use adwords than they ever managed to woo me...

Anyway, I'll sign off there. Good evening all!

Thursday, December 15, 2016

Wrangling downloading my own blog articles

99.9% of everything at the moment appears to be wrangling basic data ingest.

In a fit of semi-directionless curiosity, I decided to try expanding on my previous post by wiring up some kind of automatic blogging tool just to see if I could. This problem has many aspects. Let the yak shaving begin.

The first thing I did was start thinking about requirements a bit. I didn't, you know, write any down. But I did think about it.

I decided I needed an AI bot which could automatically write blog posts for me. I assume many people have tried and failed, or possibly even tried and succeeded, but I didn't want to let knowledge get in the way of the outcome and just ran at it.

Here is my design:
  -- A module for downloading input source data to feed into the AI
  -- A module for running learning jobs based on source data
  -- A module to write blog articles
  -- A module to publish / print out the blog articles

I thought I'd wrangle some kind of thing which would create text of the required length using some kind of minimal implementation like a markov model trained on my old blog posts. I should have just trained it on gutenberg text or something, because it turns out that downloading my old blog posts is Harder Than It First Appears. Not really difficult, but longer than the hour or so I thought it would take.

There are basically two options: web scraping or the blogger API. The tool du jour for scraping in Python is called scrapy. I decided to use the blogger API, not because I thought it was inherently better than scraping, but rather because I expect I'll get a reasonably structured object in memory at the other end and won't have to do a lot of page interpretation. Also, if I want to publish directly to blogger later down the track, it will probably use the same mechanism.

Possibly because I'm bad at searching and terrible at web programming, this took me ages to get going. In fact, I haven't gotten going yet. I have merely jumped the first couple of an unknown number of hurdles. Programming is basically like playing an infinite scrolling computer game of randomly varying difficulty until you get to the end of the current level.

The plan was to create a minimal implementation which could then be bootstrapped into a smarter implementation. I thought people might like seeing it come together (I'd open source it, and then write about the process of building it here).

Here's where I've gotten:
  -- I've installed the requisite API
  -- I've discovered that I don't just need a 'project key', I actually a full Oauth2 secret and key
  -- I read the source code to figure out that the filename that is supplied into the sample tool method in the library is actually only used to derive a directory to look for the "client_secrets.json" file, so I can actually supply a fake filename in order to tell it which directory to look for the file in. It should have just asked for a directory.

So, instead of a really cool Markov model, instead I have gotten three lines into reproducing the online walkthrough. Only took a couple hours...

More later! Wish me luck...

Wednesday, December 14, 2016

A thought experiment on making money from blogging

For some unknown reason, Google picked today (or, presumably, a few days ago) to verify my postal email address. This is somehow part of being able to accept AdSense payments. I turned ads on at some point in my blogging life, probably about five years ago.

How much was waiting for me? Had I just won the lottery? Well, it looks like Google is willing to pay me $14.68 for my five years of highly varying blog posts.

The rest of this post is just my train of thought around whether it's actually possible to turn blogging into a sensible time investment based purely on basic text ads. If I scroll through my post history, it looks like a typical number of views, or impressions, or whatever, is two to three thousand per post (total ever). My posting history is pretty random, sometimes coming as part of a coherent and interesting series of posts, sometimes totally one-off, sometimes low-quality.

One of my best post early on was when I asked people not to read it to see if I could get a baseline for how many nonhuman readers I had (bots, crawlers etc etc) because I was curious how many real people viewed my blog. This, obviously, had humans clicking furiously to find out why they shouldn't be clicking it. I should have seen that one coming! However, this blog has been gaining in popularity despite my barely posting, and it no longer stands out. I expect the general topic areas I post in are gaining wider interest.

My best ever obtained 4080 views, and comprises a now out-of-day (although not badly so) install guide for some Python machine learning tools.

The adsense rates I see right now are 22c per thousand page views, or 15c per thousand impressions. I'm a bit unclear whether those can be added together or not (e.g. say 37c per thousand visitors) or if they are different ways of tracking the average monetisation of my posts, expressed as a rate per thousand. I think the latter. So, crudely, I think if I can get 1000 of "whatever blogger counts up against each post" I will earn 22c. My average post is then work 66c. (Australian dollars)

If I could increase my view rate tenfold, I could get $6.60 per post. A post can take me anywhere from 30 minutes to ten hours to produce, depending on how much technical work is needed to underpin it. I don't do this for the money, obviously, but rather just to write up and share my own work.

To me, I think my readership likely consists of people who know me, have seen me speak, or are discovering posts based on topic keywords (such as via search). I suspect I could achieve a tenfold increase if I followed some basic tactics and invested 3 hours a week into the blog. I think that would represent achieving a larger readership within my traditional demographic by increasing both awareness of my blog, and increasing its reputation. This would be a mix of increased views per post, and increased posts per week, but still for the baseline time investment of 3 hours a pop.

If I could increase the readership an additional tenfold (100 times current), then I could haul in the princely sum of $66.66 per week. As my significant other said, I should probably start by taking a packed lunch instead. Still, that wouldn't be nearly as interesting!!!

I suspect that to increase the second tenfold would require a demographic breakthrough, or a major reputational increase. I would probably need to start investing time in researching what people think is popular, and generally speaking work for my money rather than just posting whatever's on my mind. I suspect it would be possible. I can conceive of multiple posting tracks which would spread interest more widely, including:
  -- More compelling technical content
  -- Commentary and trend analysis on technology
  -- Summaries of news and recent events in technology
  -- Writing of short fiction and stories (I dabble. I'm not good, but I suspect it would still have a positive impact on the numbers)
  -- Posting of automatically-generated content from a home-grown AI (no, I haven't built one yet, but I suspect it could be done. Again, not done well, but enough to positively impact the numbers)

Those various tactics would probably be sufficiently self-interesting that (assuming I had the spare time), I would basically enjoy the process. By this stage, the nominal visitor count would be 300, 000 per view. That sounds like a lot to me. Perhaps enough to make me question whether my reasoning has been at all correct to this point.

I spent some time reading various "start a blog" articles to try to validate what I'm thinking. The numbers just don't seem to add up based on what I'm reading. Some people seem to struggle to get 300 views per post, which is probably right for a new blog on a niche topic. Some people say getting on the front page of a major news site nets them about 300k views. All the moneymaking advise it to avoid ads and focus on things like product sales and other avenues that sound like a lot more work than doing basically nothing other than posting.

What is the experience of others? What numbers are people seeing in terms of visitors currently? Should I focus on the number of views/impressions gathered in the first week of the post only? Is my current view rate per post high, medium, or low?

Is this post at all interesting to others?

If I could achieve a third tenfold increase up to 3, 000, 000 views per post, then I'd be starting to haul in an amount of money that would start to be motivating in its own right.  Right now, that seems like a ludicrous number to consider. At that kind of view rate, I would probably have to be substantially more concerned about what I actually said, just in case someone decided to listen to me or be seriously influenced by the content. Does anyone out there get 3m views per post without massive time investment?

I'm not convinced I have been totally clear in this post. However, in the interest of maximising my return-on-time-investment, I'm *not* going to go back and tidy up! It's been about 25 minutes of typing and thinking, and I'm worried about over-investing into that 66c return!

Tuesday, August 16, 2016

Towards your first iPhone app, in Python, with some working Phone functionality

Writing your first iPhone app, with Python

Mobile apps are awesome! Writing them is super-intimidating.

This post is going to outline how to go from not having anything relevant installed, and no accounts set up, assuming no particular knowledge beyond Python and general development. I’m expecting this will involve jumping through a few hoops. I expect you’ll need to be on OSX. I’m not going to try to streamline anything, or hide any details. That would be beneficial of course, but there is a first step of understanding the current state of affairs.

You’ll also see some sections written in italics. These hold notes from the future. As I start this, I don’t know what’s involved, and I want to preserve that. However, I want to come back from time to time to tell you things you’ll want to know.


Notes from the future: So, first up: I promise this works -- to a point. It shouldn’t take a whole lot more than say two hours to get an icon on your phone which pops up a launch screen and then exits. You’ll need some kind of OSX machine to run XCode on for the final push-button manouvre.

But first, a bit about me and why I’m doing this.

I’m a Python developer. I’ve never written an iPhone app, in any language. I’ve never written a mobile app. However, I’ve got a few things I’d really like to be able to do on my phone. It’s really non-obvious how to do integrations with the apps on my phone, because most of the things in the store are consumer-oriented and are not general-purpose. I suppose there are probably lots of things out there I don’t know about, but this post isn’t about that. I need a tool for experimentation and exploration, not an infinitely large backlog of product intercomparison work.

Back to our narrative

Okay, so where to start. I’m going to be using the “BeeWare” suite of tools and packages to do as much of this in Python as possible. As I understand it, this means I’ll be able to live in Python-land to a fairly high degree, and generally treat the xcode/iOS part of the procedure as a complex deployment manoeuvre.
The BeeWare collection of projects, applications, tools and libraries which can be used for all kinds of Python development. They provide a path forward to pushing Python onto iPhone in a way which is both (reasonably) Pythonic but also utilises the native functionality of the phone effectively.
We’ll be paying attention to the following BeeWare projects:

We’ll also be using the following non-BeeWare tools:
  1. Python 3.5: I recommend using Anaconda ( to set up and manage your Python environments, and will be assumed in this post

Wait, is there a point here somewhere?

So far, you might be getting the impression that there are quite a few steps to this process, and that maybe none of them are that easy. That’s true. You might also be wondering, given all of that, whether the end product is really going to be worth all of that effort. Will Python-based apps be as good as a native Swift application? Will it be as responsive? Should I just bite a chunk off the apple and follow the crowd? That’s certainly how I feel.
As I write this, I don’t actually know. Thanks to the magic of post-editing, however, the following section contains my notes from the future, about what I was able to build, and whether the end product is satisfactory:

Notes from the future:  unfortunately this isn’t all quite demo-ready yet. It’s clearly not far off. The bugs aren’t in the fundamental iOS integration any more, they’re in the iOS bridge implementation of Toga, which is written in Python. describes what’s going on here. Each Toga back-end has a platform-specific class heirarchy which, while written in Python, contains logic specific to that platform.

This might look like bad news, but really it isn’t. Sure, we can’t go straight to the app development. However, we can now hook into the community of Python developers to implement the platform bridges for Toga. Given more time, it was entirely feasible for me to understand, debug and correct issues in the iOS Toga bridge.

Honestly, just seeing something launch on my phone was pretty exciting. It’s absolutely possible to execute Python code directly on the phone thanks to the Voodoo magic that Russell has been able to weave, and that’s mighty impressive. The next layer of the functionality onion is the GUI, and it’s just a matter of getting there.

Procedural Overview:

So, here is the end-to-end procedure, as I understand it. This simple twelve-point plan is all you need to become a mobile app coding ninja, right? Right? Don’t run these just yet -- we’ll step through them in the sections below -- but here you can see the overview all in one glance.
0. Create a project working directory for your files
                  (e.g. mkdir ~/iphone_app; cd ~/iphone_app)
1.     Set up a new virtual environment with Anaconda
2.     run ‘pip install cookiecutter’
3.     run ‘pip install briefcase’ You don’t need to do this actually
4.   run ‘pip install toga-ios==0.2.0.dev1’
5.     run cookiecutter --checkout 3.5
7.     Stick the uncompressed stuff into the directory that got made in step 4 (see online documentation item [1] below.
8.     Write the basic Python boilerplate that’s needed (described below)
9.     Somehow do something with Rubicon, because apparently it’s not part of the template or otherwise done for you Nope, this is a dependency and thanks to pip it Just Works
10.     Do a deployment magic with Briefcase This only applies to external things you want to port in, rather than things you’re building yourself here and now
11.  Load the directory of stuff into XCode
12.  Plug in ur phone and yaaay?
13. Build up the functionality you want in your application
Well, I think it’s a bit hard to get through the entire obstacle course in one go. I’m going to chunk this up into what I hope are separable work packages.

Chunking Out The Work

We’re not just looking to push out a demo here, we’re looking to establish a development loop of (write, test, deploy, test, fix): repeat. To do that we’re going to start with a Known Good Minimal implementation, push that onto the iPhone, then walk through building up a couple of iterations of our development lifecycle.

  1. Deploy some kind of zero-order application onto the iPhone with absolutely minimal functionality
  2. Get a super-simple GUI app working with Toga in my local development environment
  3. Get the GUI app working on the iPhone through that procedure
The logical step four would be automation and streamlining. However, as stated earlier, this post is constrained deliberately to covering how the existing tools work.

Overview of existing online documentation:

Step Chapter One: Zero Order iPhone App

Here is the code (called  that we’re going to use inside for our first deployment:

import toga

print("Hello ChartyButtons")

As you can see, there’s not a lot of functionality here. However, it’s enough to result in an application that has an icon on your phone, and is runnable. We can use this to establish the end-to-end deployment process, and then use that to initialise our development loop.

Set up your Python environment:
  1. Download Anaconda
  2. Run ‘conda create -p ~/Development/envs/ios python=3
  3. Run ‘source activate ~/Development/envs/ios
  4. Run ‘pip install cookiecutter
  5. Run ‘pip install toga-ios==0.2.0.dev1

Set up your project directory (contains your files)

  1. Create a project working directory for your files                   
    (e.g. ‘mkdir ~/iphone_app; cd ~/iphone_app’)
  2. Stick the uncompressed stuff into the project working directory
  3. Run ‘cookiecutter --checkout 3.5’ ← Read the next bit before you actually run this!

Step four will prompt you for some information. This is also where you give your application a name, so you might want to think about something more exciting than FirstApp...

app_name [appname]:
formal_name [App Name]:
organization_name [Example Corporation]:
dir_name [iOS]:
bundle [com.example]:

Your app_name is like a variable name. It should be short and snappy. Lowercase, with no weird characters (emoji anyone?).

The formal_name is used as a label, but shorter is still better than longer. Weird characters are untested.

The organisation_name is intended to name the copyright owner.

The dir_name can be safely left at iOS, particularly if you’re going to be writing an app that genuinely targets multiple platforms. I used the name of my project again, and nothing bad happened.

The bundle doesn’t have a great name. It comprises reverse-ordered domain name, and is intended to provide a global namespace for your code so that no two applications can collide, and so that the organisational ownership of code is clear. You can probably type any parseable URL here. I actually have a domain name registered, so I used that. The default of com.example can probably be used for local development.

Set up your application

That last step of setting up your files using Cookie Cutter will create a nearly-but-not-quite-correct XCode project for you, with a stub location for your Python-based application functionality.

Screen Shot 2016-08-16 at 11.36.54 AM.png

I’ve called my app ChartyButtons. You can call yours whatever you like, of course. This image shows the layout you need. The default template expects you to create, The README incorrectly states these should go one level too high, in the AppName/app directory. These files actually need to be in AppName/app/AppName. You’ll also notice my screenshot has a ‘.git’ directory showing in it, which isn’t shown by default. It’s not relevant to this tutorial, but it’s what I use to keep track of progress and store versions of my code that I can go back to as needed.

The file “” is the hook between iPhone-land and Python-land. This is where you will initiate your application logic. For now, just put this into (no other lines required)

import toga

print("Hello ChartyButtons")

Deploy Your Application

You now have something that can be deployed onto your phone. You need to jump over into XCode, which is an application every OSX machine should have by default. You might as well go ahead and plug in your iPhone at this stage too.

Fortunately, this is the first time I’ve ever actually run XCode on purpose, so I should be able to capture all the things which a new developer might stub their toe on.

  1. Open your project directory in XCode
  2. Unlock your phone and push “play”
  3. OUTCOME! Your app is now on the phone!

The First Time Through:
There are a few things which seem to only happen the first time.

Push “play” and wait a couple of minutes for stuff to happen in the background. It looks like XCode needs to do some compiler black magic in the background. You’ll see something like this in a status bar at the top of your screen:

The status bar should now look like this:
Once this is all finished, you can push “Play” again to initiate the deploy to your phone.

At this point, you might see something about needing to set up your Apple ID in XCode:
Hit “Fix Issue”, and type in the credentials for your Apple ID. Then push “Play” again...

The first time round, you need to physically go into your phone’s settings, and accept your own developer certificate. XCode will show you the instructions for doing so, as pictured above.

Okay. so now when you push play, you should see an icon on your iPhone representing your app.

Step Chapter Two: Get a Super-Simple GUI app working locally using Toga

Okay, so you don’t really want to involve a physical iPhone in your fast-feedback development loop. The promise of Toga is that the application can be developed entirely in Python-land if you want to, and the iPhone compatibility is just a part of the last mile deployment.

So I will start by writing up a short tutorial on building a simple Toga app, but one which has some real functionality. My app is called ChartyButtons, and the purpose (for now) is going to be to display some important charts and graphics that I want to be able to reference easily.

Notes from the future: so, the first thing I did was write up a single-button application. The button did nothing, but it was there. Russell suggested I try getting that onto the iPhone. That’s when I discovered that the iOS Toga backend wasn’t a nicely mown lawn. I as got closer to it, I could see it was in fact tall grass, hiding spiders, snakes and the occasional piece of rusted earthmoving equipment. It obviously worked once, but that time is not now. So this section really doesn’t tell the whole story.

Here is the source code for a simple, single-button GUI app in Toga

import toga

def build(app):

   container = toga.Container()

   button = toga.Button(label='DataVsAlgo')

   return container

if __name__ == '__main__':
   app = toga.App(name='Charty Buttons', app_id='blue.neuron',   

You can run this, and up pops a single button.

This is as far as I got during the time available.

Step Chapter Three: Get that App onto the Phone How to get into toga-iOS bridge development

Notes from the future: this work got a little crazy. There were dragons, line numbers, XCode debugging and branch merging. Russell got involved. We fixed stuff to make it less broken.

I have basically run out of time to write this chapter.

There’s a tool called “briefcase” which automatically packages up dependency packages into the XCode framework, but when things get hairy, you’re going to want to do this stuff the old fashioned way.

Getting your Toga app onto the iPhone properly is a little harder. The initial example didn’t make any calls to Toga, which meant that you never really needed any of its functionality.

  1. Set up a place in your development folder to store source code versions of the Toga stack
  2. Use symlinks to add them to your XCode project exactly according to the following screenshot

That is to say, you need to symlink from the app_packages directory directly to the package directory inside the containing directory for the source code. If everything were stable and “known good”, this is exactly what the briefcase BeeWare tool is intended to do for you. However, I recommend staying in charge for the time being.

This tutorial unexpectedly ends here. The future is now. For the next exciting installment, consider getting directly involved in the Great Mission! Thanks for reading this far!