CCP and customer support

A while ago Eve Online released a convenient ‘launcher’ for their game which I’ve played on and off for almost a decade. Except it wasn’t very convenient, it tried to wrap the update process and from the moment it was released it has never worked properly and I’ve experienced the same bug with it since day one resulting in a highly tedious process to update the client. Sadly I only have some ineffective logs to go by so I can’t speculate on why it fails every single time, but I filed bug report after bug report with as much information as possible saying this was seriously impacting my ability to play. I reported it on their forums to no avail, and I filed support issues which were never replied to. I found workaround after workaround (which I also posted on their message boards for others) and then my last workaround to the broken launcher stopped working too, I could no longer play the game I was paying for so I gave up and I cancelled my two subscriptions.

During cancelling my subscription I was asked for a reason, so I wrote pretty much the above account. All I got in response was a mail reminding me to resubscribe, and then (in reverse order)…

From: Colin Alston
Date: Sun, Nov 3, 2013 at 1:23 PM
Subject: Re: Re: EVE Online - Subscription Renewal Reminder 
To: EVE Online Customer Support <>

Your reply isn't "Can we fix our launcher and get you back as a customer"? Wow...

On Sun, Nov 3, 2013 at 1:20 PM, EVE Online Customer Support <> wrote:
Hello, Senior GM Huginn here,

I'm sorry to hear that you're still experiencing issues with the EVE launcher.

What are the user names of the accounts you want us to cancel the subscription for?

Best regards,
Senior GM Huginn
CCP Customer Support | EVE Online | DUST 514
Original ticket @ 2013-11-03 11:19:

Your records should indicate that I deliberately cancelled my subscription
because CCP aren't interested in fixing their broken launcher which I've
filed multiple bug reports about.

On Sun, Nov 3, 2013 at 12:03 PM, EVE Online support

> [image: EVE Online] 
> We are contacting you to remind you that according to our records you have
> 6 days left on your non-recurring subscription to EVE Online.



SSH ports, the great obscurity debate

So there are two posts and

Well clearly we have some very different arguments, which are of course all nonsense. Security by obscurity is still valid security, as long as it isn’t your only security – that much is certainly valid. As to what port SSH runs on, it doesn’t matter in the slightest if your SSH daemon is insecure, it’s outright trivial to know if an SSH daemon listening on any port. What people seem to fail at most with security is that it hinges on the principal of risk, and with so much software out there people are overwhelmed and paranoid about all these cumulative risks causing them to make rash decisions which don’t improve security but make it more difficult to work with those systems. Reality is that there has not been a reasonable SSH exploit in the last 15 years, other than the great Debian faux pas with ssh key generation.

Personally I can’t be bothered to move SSH to a different port, it only provides a minimal level of cover for the unlikely possibility of an SSH zero day but brings with it a substantially greater amount of inconvenience. What bothers me more is that both these articles fail at explaining how to really secure SSH, since if your daemon is open to the public internet in any form then you’re already in a difficult position. This is simple enough though to achieve a reasonably high barrier to the most common means of exploitation – enforce the use of sudo for root access and the use of SSH keys for login.

PermitRootLogin no
PasswordAuthentication no

Of course if the keys to accounts are not secured by users in some way (their machines are exploited) then you’re quite screwed no matter what you’ve done. If you require security over and above that then a bastion host is a reasonable option with SSH traffic restricted to that IP address, and/or providing a VPN with two factor authentication into a DMZ with SSH access (you can do this with OpenVPN).

Security is about a wider scope of architecture, and changing service ports is a serious waste of time, as is even debating something so silly. The only secure machine is one which is not powered on.

Amazon Auto Scaling with Puppet, PuppetDB and Haproxy

In a previous post I  talked about how we bootstrap EC2 instances into Puppet with an rc.local script inside a stock AMI, this has worked great but turns out it’s somewhat difficult when using something like Auto Scaling groups. Auto Scaling sadly provides very little metadata to work with, the entire idea really hinging off preconfigured AMI’s – something I absolutely hate. The other issue with that is you’re in the same place trying to organise hosts off the bat by their hostnames.

After days of researching all the angles I finally stumbled on a presentation done by Pinterest (people like Ryan Park who publish their ops work are seriously awesome) which had some clues about evolving my rc.local setup. A while ago I changed our AMI to rather fetch the bootstrap script from a webserver on the Puppet master and keep it in the same repo as our Puppet modules, this saves rebuilding the AMI if we need to change the Puppet host or anything like that – or in this case adding crazy hacks. This is kinda similar to how the Pinterest stuff was handled, with the exception of not requiring the ec2 utils which would put keys inside an AMI and that would make me pretty uncomfortable.

So the first step is a new bootstrap script called by an rc.local of wget -O /tmp/; bash /tmp/ or some such, obviously wherever you host your scripts could be totally different.

id=`/usr/bin/curl -s | cut -c 3-`
FQDN=`/usr/bin/curl -s | grep hostname | sed 's/.*=//' | sed 's/ //'`
IP=`/usr/bin/curl -s`
FQDN=`eval echo $FQDN`
HOSTNAME=`echo ${FQDN} | awk -F"." '{print $1}'`
if [ "$FQDN" == "" ]; then
   echo "No hostname found in user metadata"
   exit 0
echo $FQDN > /etc/hostname
cat<<EOF > /etc/hosts
# This file is automatically genreated by ec2-hostname script   localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
hostname $FQDN
# Install puppet
dist=`lsb_release -cs`
echo "deb $dist main dependencies" > /etc/apt/sources.list.d/puppet.list
/usr/bin/apt-key adv --keyserver --recv-keys 4BD6EC30
/usr/bin/apt-get update
/usr/bin/apt-get -y --force-yes install puppet
wget -O /etc/puppet/puppet.conf
/usr/bin/puppet agent --onetime --no-daemonize --logdest syslog
echo "#!/bin/sh -e" > /etc/rc.local
echo "exit 0" >> /etc/rc.local

What manner of witchcraft is this? Well we grab “hostname=blah” from the user-data field on an EC2 host, but notably we evaluate it in the context of this scripts scope, then we get the IP, build a hosts file and set the hostname up right, and then it goes through the motions of installing puppet and kicking off a manual Puppet agent run (these days I cron Puppet, running the agent as a daemon sucks) and then finally blank out the rc.local script so we never run it again.

Now when we setup an Auto Scaling group, using a VPC subnet (you should use VPC… it saves a pile of headaches).

$ as-create-launch-config vpclc --image-id ami-YOURAMI \
    --instance-type m1.large --region eu-west-1 --key YOURKEY \
    --user-data 'hostname=prd-web-${id}'
$ as-create-auto-scaling-group vpcasgroup --launch-configuration vpclc \
    --availability-zones "eu-west-1c" --min-size 1 --max-size 5 \
    --desired-capacity 1 --vpc-zone-identifier "subnet-SOMEVPCSUBNET" \
    --region eu-west-1 --tag "k=Name,, p=true"

Yup! We stuck a piece of shell script in the user-data, and now the bootstrap script will dynamically replace it with the unique part of the instance ID that we got from the EC2 API and still work with other hosts too.

Of course out in the real world where things are almost never done properly and never really in our control, using ELB is unfortunately a pain in the ass. Half the time I ask people to add a CNAME to one of our servers, a load balancer, etc, they dig the IP address and add an A record – *sigh*. On top of this a CNAME can’t exist on a domain apex, which throws ELB out the window entirely. Route53 can deal with this, but if you think a giant corporate is going to delegate me their entire domain name you’re dreaming.

Enter haproxy, an amazing piece of software that we use in a thousand places at Praekelt. So the question is, since there’s no DNS above (adding automatic updates to Route53 is something I have yet to bother with) how to get these into a load balancer without using ELB which can update dynamically with Auto Scaling. Well, using PuppetDB like I also previously wrote about.

A sample init.pp

class loadbalancer {
   $myhosts = query_nodes('hostname~"prd-web-', ipaddress)
   file {'/etc/haproxy/haproxy.cfg':
      ensure  => present,
      content => template('loadbalancer/haproxy.cfg.erb')
   service {'haproxy':
      ensure    => running,
      subscribe => File['/etc/haproxy/haproxy.cfg']
   package {'haproxy':
      ensure => latest

Sample haproxy.cfg.erb

listen webstuff
   bind *:80
   mode http
   option httpchk GET / HTTP/1.1\r\nHost:\
   appsession JSESSIONID len 32 timeout 3600
<% myhosts.each_with_index do |ip, i| %>
   server web<%= i %> <%= ip %>:80 check port 80 weight 1 maxconn 1500 inter 10000
<% end %>

And there it is, a rough construct of using Auto Scaling somewhat sanely with Puppet, and your own load balancer.

Open Source in government

On several occasions I’ve seen the Open Source community (more usually, those fanatical about the concept rather than authors within the space) petition for government to stop spending money on software licenses. At its core this is a good idea but along the way I’ve witnessed many do’s and don’ts with this approach. What has worked for me, and gotten not just several bits of open source into government systems but the funding to do it, is the ethos I document here.

Rule number 1: Don’t be a bully

And it’s the only rule. Don’t be a jerk. Yes, it’s your TAX money, no one cares, but the government does actually want your help, and in my experience they will take it every time – but some people there are just as afraid of losing their job as anyone else out there, and extremely afraid of being made to look incompetent. Don’t tell them what to do, guide people to your way of thinking. Get people in your corner and then run with it. Lobbying and making a huge fuss right out the gate will make people oppose you and think you’ve got some agenda, but if they think you’re going to make them look good – you’re golden.

Do NOT try to push Linux on the desktop…

The first thing people want is for every government workstation to run Linux. As someone who writes open source software, and works with it every day, I don’t use Linux on my workstation for a bunch of reasons mostly out of my control. The first is that I usually work on my MacBook, and OS X is great. The second is that X doesn’t cooperate with my desktop. I’m sure someone with more patience could hack it to pieces and make it work, I’m sure I could if I could be bothered – but I don’t, because I actually still require Windows for a few things like playing games and managing servers with their really poorly written management interfaces, one of which only works on IE8. My point is this, attacking the Windows desktop will get you shot down.

   … Because compatibility

Through many years of “my nephew James knows computers” built systems which are only compatible with IE6, Adobe based forms systems, Access databases, VBScript tied into Excel based tools, the list goes on. As someone who spent many years working with government to design and implement open source systems (yes, they really do use some stuff already), these nightmarish systems that most of us only read about on The Daily WTF exist in abundance. We’re talking about the stuff of nightmares here, and very solidly entrenched stuff.

… Because cost

If you know OEM’s you’ll know that somehow they can build cheaper computers than it seems we can build ourselves when paying out our nose for a standalone copy of Windows. Genuinely through various schemes, subsidies and discounts I can tell you that government pays extremely little in the grand scheme of things for licensing Windows. It’s a nominal cost, made more nominal through Microsoft bulk licensing programmes. It would cost them a whole lot more in man hours to replace Windows on their desktops than it would to just continue using it, never mind re-training a few million staff members who are less technically competent than your grandmother.

Build it and they will come

I propose a different tactic, focus on the backend and people facing systems. There is benefit to open source in government, but it’s open source between government. The UK for example has a pretty good system for managing drivers licenses – but South Africa’s is terrible, they spent billions getting ‘eNaTiS’ running and it is a pathetic failure. That’s not entirely the governments fault, it’s also because of the incompetent fraudsters who claim they can build these systems in the first place – the Johannesburg municipal billing crisis being another great example. And then there’s the DNA sequence databases used around the world for crime fighting – why the hell are they all proprietary monsters that can’t talk to one another? Do we not want to solve crime? Why couldn’t we just go “Hey, England, how about you give us a copy of that system you’ve got”. Why did we have to pay Germany 2 billion euros to get a license plate recognition system? Well, it’s not as if there’s an open source system for that.

Problem is we can’t just tell government they have to use open source, come up with a list of reasons why, and then not actually show them how. Government represents an arbitrage opportunity to these dishonest cowboy businesses which have never done anything before, and then disappear into the night leaving increasingly terrible systems in their wake, those are the people we really need to fight and publicly denounce. So, pitch in a tender with an R1.00 price tag if you want them to go open source. These are actually fairly easy systems to build if they’re well thought out, but we first need the right people to start building them.

If we investigate the requirements of these systems and build well designed open source alternatives, then we can actually start talking about asking our governments to use them. I think at that point though we won’t have to do much talking.

Why touch screen information boards annoy the heck out of me

We’ve all seen these things lately, right?

Woolies Store

No, not people shopping at Woolworths. I’m talking about the giant white obelisk like thing poking out of the floor that everyone has to navigate around. These touch screen information directory nightmares which “help” you find a store, I hate them and I wish they would die, and I wish the people making them and selling them to shopping centers would do something useful for humanity, like jump off a bridge.

1. They are trying to solve a problem which did not exist
And now their only purpose in life is to keep the companies which make them in business. A cheap piece of paper stuck onto a board with shop labels and an index of shop names was more than simple enough. It worked fine. There was a sticker showing you where you were, and a map to show you where you were going. If that was honestly too complicated for you, then I’m not sure how you arrived at the store, or how you obtained money to purchase anything from it, and I suggest you visit an optometrist.

2. They’re an invasion of privacy
Sure that’s going to sound nuts at first but think about it, these things are massive, bright, and have crazy huge fonts. So the stores you’re looking for are getting printed for anyone to see. That’s not necessarily dangerous, but it could be under some weird circumstances. In any event, I’m uncomfortable with the fact that someone I don’t know knows where I’m going in a big shopping center. They could follow me to the bank, or to a jewellery store – they could do that anyway, but all they need to do now is hang around the information board to pick their victims.

3. They don’t work without electricity
So neither do most shops these days, but still… they needlessly waste electricity.

4. Only one person can use it at a time
If there was some confused sod at the old dead tree of store locations, you could just lean over their shoulder and invade their personal space until they left or you managed to see what you needed to see. Now you have to wait for the guy to finish searching through the worlds worst case/hyphen/apostrophe sensitive indexing system to find what he wants – or worse browse some colossal, and stupidly specific, nested index of store “genres”. Hmm, is Dion Wired under “electronics”, “home”, “gaming” or “appliances”? That sort of varies based on why you’re going there… Perhaps the guy in a hooded jumper lurking around can help.

5. They’re almost never calibrated correctly
Seriously, every one I’ve ever tried to use had the touch panel so poorly calibrated you had no idea where to press, or you basically had to smash your fist into the thing to get a response, or the software was just terrible. Plus I’m sure after a few thousand people use them they’re ready for the trash.

Just what on earth was wrong with having a sheet of paper? I’m a tech guy, but seriously if it ain’t broke…

(Any reference to Nicolway center is coincidental, it’s a cool place, and these damn things are almost everywhere)

How I got started

Peter wrote a cool post about how he got started in his career, it seemed like a good idea because people seem to ask about this kind of thing but unfortunately it’s not a reproducible set of instructions unless you’re willing to invest over 20 years of time and passion.

So I guess for me it all really started around 1992 with a 386, my fathers work computer which he would occasionally bring home. I was about 7 or 8 years old and it had a game called Nibbles and another called Gorillas which I used to play against my sister.

In truth even before this I had a fascination with how things worked, most toys I got were dismantled to find out where the artificial smoke came from inside the toy train – and then things like attempting to make it come from my stuffed toy crocodiles nose. Most of this failed, but in fact it was my grandfather (I knew him as “Oupa” because my mothers side is Dutch) who noticed this and provided me with odds and ends he’d bought at auctions for the sole purpose of me ripping them apart to stop me dismantling the family TV set.

It didn’t take me long to figure out how to start the old Gorillas game up on my own, and then that it was written in QBasic. I started slowly figuring out that these mountains of files controlled how the game worked, and started altering them to make my bananas more explosive than my sisters. I spent a lot of time in libraries reading every book I could understand on the subject of programming, computer architecture and electronics.

But I guess it was my total disregard for the negative effects of messing with things I don’t understand which was my gift, I broke a lot of stuff as a kid, especially computers, and was punished for it frequently to no avail – I told myself “They just refuse to understand that I need this like they need oxygen”. Most adults then saw it as a curse but instead of becoming a serial killer I became a hacker, and to me the first time I connected to the internet was like someone giving a normal child their own candy store – infinite choice.

SA exchange controls and why you really should care

Even if you’re not a South African you should still care about our exchange control regulations, because this is a great country and we need investment – unfortunately, we can seldom afford to accept it. Currently Mark Shuttleworth is involved in a court case against the SA government over exchange control regulations. In typical style, the government officials are overplaying their hand claiming that if he wins then the country will be destroyed – this could not be further from the truth, as a matter of fact what most of the public don’t know (since these rules don’t affect the majority, and are hard to understand) is that they could have the opposite effect, and further still the current regulations have actually crippled SAs economic growth for decades.

What the media doesn’t entirely point out is what these rules actually are. First off, no individual (anywhere in the world) can exchange more than 4 million ZAR in their lifetime, or more than R1mi in offshore funds. Companies also need to motivate any transfer of funds and get approval, which is undoubtedly slow and often refused. While this might seem like a lot of money to most people, in reality it isn’t and these numbers are never maintained in-line with global inflation.

The effect of regulations like this is very similar to transfer duties on houses. The average cost of a very modest house or apartment in my area is around R1mi (+/- $100,000), however the cutoff is R600,000. Feel free to check out any property website and see what you can get for that – nothing. For me to buy the property I rent, in addition to paying for the property I’d be forking over R100,000 to the government for absolutely nothing in return and no work on their part – I still have to pay rates, taxes and everything else for as long as I own it, and I still had to pay taxes to earn that R100,000. This means a lot of people who can actually afford to buy can’t buy, and the government is too incapable and slow to update these rates in line with actual economic conditions.

So, back to exchange controls. Now you’ve made a bunch of money, but despite the fact that you’ve paid taxes and VAT and UIF to all the people who can’t get jobs, and E-tolls so that SANRAL can ship R20bn off to Germany without impeachment – you decide “Yeah, actually, screw this place” and you head off somewhere like Shuttleworth or Musk did. Now suddenly your fortune has to remain in SA because of exchange control regulations, or you’re taxed to high heaven on the income you were already taxed on when you earned. Now you may think that this is good news, these rich guys can’t just take all the money out of our economy and leave it to die. That may be one consequence, but consider the other – who in their right mind is going to bring money into the country if they’re never allowed to leave with it, and what incentive is there for anyone to even bother running a big business in SA?

There in lies the rub, there isn’t enough money to go around in SA for investment in growing our economy – yet the government says we’re not allowed to bring any in, or invest any out. No jobs for you then, unless we can fund companies out of thin air without foreign investment. Even if you run your business from South Africa, it’s too difficult to do business with the rest of the world thanks to the same regulations. So naturally the smart South Afircan innovators setup their businesses outside South Africa where they’re more welcome, more likely to succeed and find talented staff, taxed way less, able to garner investment and start-up capital more easily, and most importantly – free to leave and spend it anywhere in the world, to the point where London is littered with companies run by former South Africans, money which could be here building houses and creating jobs.

Using PuppetDB for template generation

For quite some time it has bugged me that Puppet didn’t have a way to query information about neighbouring servers, something which is extremely handy for generating configurations which need to know about the other servers in your environment (Munin configs, firewalls etc). A few things filled this gap but I always felt them lacking in many respects, as I write this I can’t put my finger on what the reasons were.

The addition of PuppetDB provides a really good solution for this though. Installing is quite trivial, for our environment we don’t have too many servers and the Puppet master is quite large so I just installed everything on there.

Once everything is working right you can access the DB directly, or use the useful API to query things about your nodes. The database will be populated with nodes as they checkin (do configure with PostgreSQL per the documentation).

puppetdb=# select count(*) from certnames;
(1 row)

The next thing I wanted to do though was be able to generate a template from a list of hosts. For this I found the very good puppetdbquery module by Erik Dalén.

The installation details are a bit sketchy so here is how I set it up. Firstly just ‘gem install ruby-puppetdb’ on the master, if all went well you should now have a ‘puppet query’ command which can pull hosts from PuppetDB based on facts and other information.

~# puppet query nodes 'Package[nginx]'
["web01", "web02"]

Now we just need this exposed in our modules. Assuming you have pluginsync enabled on your agents, in your puppet tree create a modules/puppetdb folder and copy the ‘lib’ folder from the puppetdbquery repo into there.

~$ mkdir -p /etc/puppet/modules/puppetdb
~$ git clone git://
~$ cp -a puppet-puppetdbquery/lib /etc/puppet/modules/puppetdb/

Your agents should now sync the library in, if you have any errors then depending on how you’ve configured the puppet master you may need to restart puppetmaster or Apache to make sure the library gets loaded correctly.

Now to use it in a module, here is my new Munin configuration.


class muninserver {
   package { 'munin':
      ensure => latest
   $orghosts = query_nodes('kernel="Linux"')
   file {'/etc/munin/munin.conf':
      ensure => present,
      content => template('muninserver/munin.conf.erb'),
      owner => root,
      group => root,
      mode => '0644',
      replace => true


# Munin config
includedir /etc/munin/munin-conf.d
htmldir /var/www/munin
graph_strategy cgi
# Hosts
   use_node_name yes
<% orghosts.each do |host| %>
[<%= host %>]
   address <%= host %>
<% end %>

Sexism, beyond all reasonable doubt.

Firstly, I’m a Python developer, and I have been one for well over 10 years. I personally know a number people who were at PyCon this year, and sadly all the great things which happened there got destroyed by a silly spat over jokes that were perceived as sexist/derogatory/inappropriate.

Many people who sadly probably don’t even know what Python is have picked up the issue and blurred it beyond comprehension.

Here’s what happened in short

  • Adria Richards was offended by two guys joking in the seats behind her. Instead of asking them to shut up, she methodically took a picture of them which clearly identified their employer and one of the individuals names, and posted it to her 10,000 Twitter followers with a derisive comment. Only after that damage was done did she attract the attention of PyCon representatives to deal with the issue. As it turns out, this isn’t the first time she’s been passive aggressive about this issue.
  • One of the people from Playhaven was fired because of it. Alex Reid was not the person fired, that persons anonymity remains somewhat in tact.
  • Adria Richards was then fired

Here’s why people on both sides of this issue are full of crap

  • Publicly, irreversibly, and unashamedly destroying someone’s career and possibly their personal life is an unacceptable, disproportionate and arguably illegal response to taking exception to two men having a laugh between each-other, or at the most making offensive jokes.
  • If anyone think the female gender is void of women who make jokes at the expense of men, they are completely delusional.
  • Adria was not fired for “defending feminism” or “fighting misogynists”, that is just dishonest hyperbole. There has been no statement of why she was, if I’d hazard a guess it’s because she went on to deliberately drag her company into the issue.
  • No one actually knows what Adria heard, they are jumping to conclusions in support of their own beliefs on the issue. We all know how sensitive some people can be, and equally how disgusting some people can be.
  • The jerks, misogynists, and DDoS script kiddies attacking people in response to the various comments are not part of the Python community.
  • Ignorance begets ignorance, and a strong one-sided view of this issue is only going to encourage a stronger view opposing it.

And finally, yes, the tech industry is populated mostly with men, along with several thousand other careers. Most of those men have probably had extremely negative experiences with society and women in particular, and a lot of them have much different standards of communication than what people perceive as sociable. It is not the place of those outside the actual hacker and software community to judge the inner workings of it or the social skills of the people who built it, suffice to say there are many women I’ve met who have no problem with it at all.

It should also be pointed out that one incident doesn’t make a pattern. People jumping on the bandwagon saying the men involved in this incident were “sexist” is about as accurate as calling me an anti-semite because you heard me joking with a friend about the fact that he’s a Jew, or calling someone a racist because they said something that could be construed as racist. Reality is we all think these things sometimes, and we all say these things sometimes without thinking it through, and every single person is capable of social faux pas. The real shame here is that two people lost their jobs because they were ultimately only guilty of being human.

Regardless of what may have been said on that day, the result of hypersensitivity is always the same outcome, and it would be extremely unfortunate and disappointing for people to turn hacker conferences into boring, politically correct corporate affairs instead of light hearted and free-form exchange of ideas which is not based on judging peoples personality but rather their actual ideas, as most of these people have contributed a damn sight more to society than the treatment or reward they’ve received from it. That warrants understanding, tolerance, and at the very least – benefit of the doubt.