One of the things we do at work is streaming video, in fact it’s a pretty big part of our business. Streaming video presents a number of problems, and monitoring is one of the biggest – people get pretty unhappy when their TV show cuts out, or they miss the winning goal in a soccer match because of some silly issue.

What makes this challenging is that it’s not all that easy to solve reliability issues with just redundancy, and there are a lot of things which can go wrong. So along with checking every component, we need to verify the end result and check every channel to make sure the audio quality is reasonable, there’s no MPEG artifacts (which can be caused by anything from a faulty encoder to a lightning storm), and that there’s actually a picture. How does that happen? People. People painstakingly check every channel every hour of the day and make sure people are seeing what they want to see.

The problem is a lot of quality issues are subjective, and people make mistakes so my job is to automate quality control and measurement, and generate alerts when things go wrong. But how on earth is this possible for streaming video?

The first thing is to look at what video is – it’s a stream of data, or on a lower level a multi-dimensional matrix of pixels streaming through time. Of course analysing this is a huge job for a computer, especially with a few hundred channels and endpoints, so you need some dimensional reduction.

The histogram

If you’ve ever used photo editing software you’ve probably seen something called the histogram. What it represents is the frequency of colour intensities in the image, which is useful for detecting when videos plummet into darkness.

Looking at a histogram, the things we do want and don’t want are immediately obvious but how does this look with an extra dimension, namely time?

Taking a TV show and smoothing out a histogram for each frame and plotting that as a surface to see what’s going on turns out like this.

Pretty, but still too many dimensions to monitor real time (fortunately I have a monstrous workstation to generate these graphs). Source here.

Obviously what we don’t want is the histogram leaning heavily to the right or left, but more importantly we expect it to jump all over the place – ie, have a high variance. So the first thing we do is reduce the dimension by taking the sum of pixels and calculating the percentage of each luminosity value. Then we sum the multiple of those and arrive at just a value of how light or dark each frame is. This gives us a value from 0 to 255 which indicates where the histogram peaks, which ideally is not near the extremes if there’s a visible picture and it’s not roasting peoples eyeballs.

You can clearly see the value drop off when the credits roll, but there’s still variance which gives us a good idea that something is going on. Snow, test patterns or black screens will have a very low variance.

Motion detection

Tracking the variance in the histogram gives us an idea that there’s something resembling a video, but another good indicator is some simple motion detection. We can get a matrix of changed pixels by calculating the difference between frames.

1
2
3
4

def frameDiff(t0, t1, t2):
    d1 = cv2.absdiff(t2, t1)
    d2 = cv2.absdiff(t1, t0)
    return cv2.bitwise_and(d1, d2)

More here.

Sending it all to Riemann

Using Python and OpenCV this is what we end up with the following functions which takes the 95th percentile of motion changes as a percentage of the visible area, and returning the mean and variance in the histogram value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

import cv2
import numpy
 
 
def frameDiff(t0, t1, t2):
    d1 = cv2.absdiff(t2, t1)
    d2 = cv2.absdiff(t1, t0)
    return cv2.bitwise_and(d1, d2)
 
def processStream(chan, fname):
    cap = cv2.VideoCapture(fname)
 
    bright = []
    frameBuffer = []
    tdiff = []
 
    while(cap.isOpened()):
        ret, frame = cap.read()
 
        if ret:
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
            frameBuffer.append(gray)
 
            r = cv2.calcHist([gray], [0], None, [256],[0,256])
            cv2.normalize(r, r, 0, 255, cv2.NORM_MINMAX)
 
            # Calculate value
            v = sum(r)
            ev = sum([i * (c/v) for i,c in enumerate(r)])[0]
 
            bright.append(ev)
 
            # Perform motion detection
            if len(frameBuffer)==3:
                diff = frameDiff(*frameBuffer)
                h = len(diff)
                w = len(diff[0])
                avm = (sum(sum(diff)) / float(h*w))*100
                tdiff.append(avm)
                frameBuffer = []
            else:
                break
    cap.release()
 
    if bright:
        avsq = float(numpy.mean(bright))
        sqvar = float(numpy.var(bright))
        vmotion = float(numpy.percentile(tdiff, 95))
    else:
        avsq, sqvar, vmotion = 0, 0, 0
 
    return avsq, sqvar, vmotion

Unfortunately I can’t show you my Tensor plugin for this because there’s a bunch of secret API and stream source logic wrapped up in it. But lets pretend a video stream comes from somewhere in some format without any DRM, gets buffered in small chunks in a big ramdisk every few seconds and then gets sent to the functions above and lands up in Riemann and consequently InfluxDB.

So what happens when things go wrong? Well, at least one of these tends to flat line. The variance of the histogram goes to zero when there’s no more change in the histogram of the frames, this generally means you’re locked on a static image or stuttering between frames – this is more useful than the histogram value itself because beyond a fairly small threshold no particular value indicates an issue – otherwise you’d trigger an alert every time credits roll, or miss something like a test pattern coming through.

Lurking and occasionally trying to help people in #twisted has taught me one thing – no one ever reads the documentation, I mean really read it cover to cover. Seriously, if you’re not used to all the asynchronous design patterns and you just start trying to bulldoze your way through writing a Twisted application then you’re going to get very frustrated and fail. The other thing I realise is that people stop at trying to use reactor.run() and building a crap pile around it. Seriously, use plugins.

But if you’re writing some kind of client, plugins are less obvious. Fortunately, writing something that implements IService is not difficult, and very rewarding in terms of clean code which is properly testable.

So lets start by creating twisted/plugins/ticktock_plugin.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

from zope.interface import implements
 
from twisted.python import usage
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker
 
import ticktock
 
class Options(usage.Options):
    optParameters = []
 
class TickTockServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "ticktock"
    description = "A clock"
    options = Options
 
    def makeService(self, options):
        return ticktock.makeService()
 
serviceMaker = TickTockServiceMaker()

Now lets write a Service, called ticktock.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from twisted.application import service
from twisted.internet import task
 
class LoopingService(service.Service):
    def __init__(self):
        self.t = task.LoopingCall(self.tick)
        self.n = True
 
    def tick(self):
        if self.n:
            print "Tick"
        else:
            print "Tock"
 
        self.n = not self.n
 
    def startService(self):
        self.running = 1
        self.t.start(1.0)
 
    def stopService(self):
        self.running = 0
        self.t.stop()
 
def makeService():
    return service.LoopingService()

Now you can run your service with twistd which takes care of logging and all sorts of other things you shouldn’t try doing yourself.

1
2
3
4
5
6
7
8
9
10
11

$ twistd -n ticktock
2014-04-13 18:50:36+0200 [-] Log opened.
2014-04-13 18:50:36+0200 [-] twistd 13.2.0 (/home/colin/ticktock/ve/bin/python 2.7.3) starting up.
2014-04-13 18:50:36+0200 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2014-04-13 18:50:36+0200 [-] Tick
2014-04-13 18:50:37+0200 [-] Tock
2014-04-13 18:50:38+0200 [-] Tick
2014-04-13 18:50:39+0200 [-] Tock
2014-04-13 18:50:40+0200 [-] Tick
2014-04-13 18:50:41+0200 [-] Tock
2014-04-13 18:50:42+0200 [-] Tick

Obviously instead of a looping call you can throw in an IRC client, DDoS your enemies, or whatever you were planning to do with it.

And yes, there is already a timer service class available in Twisted.

So there are two posts http://www.danielmiessler.com/blog/putting-ssh-another-port-good-idea and http://www.adayinthelifeof.nl/2012/03/12/why-putting-ssh-on-another-port-than-22-is-bad-idea/

Well clearly we have some very different arguments, which are of course all nonsense. Security by obscurity is still valid security, as long as it isn’t your only security – that much is certainly valid. As to what port SSH runs on, it doesn’t matter in the slightest if your SSH daemon is insecure, it’s outright trivial to know if an SSH daemon listening on any port. What people seem to fail at most with security is that it hinges on the principal of risk, and with so much software out there people are overwhelmed and paranoid about all these cumulative risks causing them to make rash decisions which don’t improve security but make it more difficult to work with those systems. Reality is that there has not been a reasonable SSH exploit in the last 15 years, other than the great Debian faux pas with ssh key generation.

Personally I can’t be bothered to move SSH to a different port, it only provides a minimal level of cover for the unlikely possibility of an SSH zero day but brings with it a substantially greater amount of inconvenience. What bothers me more is that both these articles fail at explaining how to really secure SSH, since if your daemon is open to the public internet in any form then you’re already in a difficult position. This is simple enough though to achieve a reasonably high barrier to the most common means of exploitation – enforce the use of sudo for root access and the use of SSH keys for login.

1
2

PermitRootLogin no
PasswordAuthentication no

Of course if the keys to accounts are not secured by users in some way (their machines are exploited) then you’re quite screwed no matter what you’ve done. If you require security over and above that then a bastion host is a reasonable option with SSH traffic restricted to that IP address, and/or providing a VPN with two factor authentication into a DMZ with SSH access (you can do this with OpenVPN).

Security is about a wider scope of architecture, and changing service ports is a serious waste of time, as is even debating something so silly. The only secure machine is one which is not powered on.

In a previous post I  talked about how we bootstrap EC2 instances into Puppet with an rc.local script inside a stock AMI, this has worked great but turns out it’s somewhat difficult when using something like Auto Scaling groups. Auto Scaling sadly provides very little metadata to work with, the entire idea really hinging off preconfigured AMI’s – something I absolutely hate. The other issue with that is you’re in the same place trying to organise hosts off the bat by their hostnames.

After days of researching all the angles I finally stumbled on a presentation done by Pinterest (people like Ryan Park who publish their ops work are seriously awesome) which had some clues about evolving my rc.local setup. A while ago I changed our AMI to rather fetch the bootstrap script from a webserver on the Puppet master and keep it in the same repo as our Puppet modules, this saves rebuilding the AMI if we need to change the Puppet host or anything like that – or in this case adding crazy hacks. This is kinda similar to how the Pinterest stuff was handled, with the exception of not requiring the ec2 utils which would put keys inside an AMI and that would make me pretty uncomfortable.

So the first step is a new bootstrap script called by an rc.local of wget http://puppet.acme.com/ec2-bootstrap.sh -O /tmp/ec2-bootstrap.sh; bash /tmp/ec2-bootstrap.sh or some such, obviously wherever you host your scripts could be totally different.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

#!/bin/bash
 
id=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id | cut -c 3-`
FQDN=`/usr/bin/curl -s http://169.254.169.254/latest/user-data | grep hostname | sed 's/.*=//' | sed 's/ //'`
IP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
 
FQDN=`eval echo $FQDN`
HOSTNAME=`echo ${FQDN} | awk -F"." '{print $1}'`
 
if [ "$FQDN" == "" ]; then
   echo "No hostname found in user metadata"
   exit 0
fi
 
echo $FQDN > /etc/hostname
 
cat<<EOF > /etc/hosts
# This file is automatically genreated by ec2-hostname script
127.0.0.1   localhost
$IP  $FQDN $HOSTNAME
 
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
EOF
 
hostname $FQDN
 
# Install puppet
dist=`lsb_release -cs`
echo "deb http://apt.puppetlabs.com/ $dist main dependencies" > /etc/apt/sources.list.d/puppet.list
/usr/bin/apt-key adv --keyserver keys.gnupg.net --recv-keys 4BD6EC30
/usr/bin/apt-get update
/usr/bin/apt-get -y --force-yes install puppet
 
wget http://puppet.acme.com/puppet.conf -O /etc/puppet/puppet.conf
 
/usr/bin/puppet agent --onetime --no-daemonize --logdest syslog
 
echo "#!/bin/sh -e" > /etc/rc.local
echo "exit 0" >> /etc/rc.local

What manner of witchcraft is this? Well we grab “hostname=blah” from the user-data field on an EC2 host, but notably we evaluate it in the context of this scripts scope, then we get the IP, build a hosts file and set the hostname up right, and then it goes through the motions of installing puppet and kicking off a manual Puppet agent run (these days I cron Puppet, running the agent as a daemon sucks) and then finally blank out the rc.local script so we never run it again.

Now when we setup an Auto Scaling group, using a VPC subnet (you should use VPC… it saves a pile of headaches).

1
2
3
4
5
6
7
8

$ as-create-launch-config vpclc --image-id ami-YOURAMI \
    --instance-type m1.large --region eu-west-1 --key YOURKEY \
    --user-data 'hostname=prd-web-${id}.aws.acme.net'
 
$ as-create-auto-scaling-group vpcasgroup --launch-configuration vpclc \
    --availability-zones "eu-west-1c" --min-size 1 --max-size 5 \
    --desired-capacity 1 --vpc-zone-identifier "subnet-SOMEVPCSUBNET" \
    --region eu-west-1 --tag "k=Name, v=prd-web-N.aws.acme.net, p=true"

Yup! We stuck a piece of shell script in the user-data, and now the bootstrap script will dynamically replace it with the unique part of the instance ID that we got from the EC2 API and still work with other hosts too.

Of course out in the real world where things are almost never done properly and never really in our control, using ELB is unfortunately a pain in the ass. Half the time I ask people to add a CNAME to one of our servers, a load balancer, etc, they dig the IP address and add an A record – *sigh*. On top of this a CNAME can’t exist on a domain apex, which throws ELB out the window entirely. Route53 can deal with this, but if you think a giant corporate is going to delegate me their entire domain name you’re dreaming.

Enter haproxy, an amazing piece of software that we use in a thousand places at Praekelt. So the question is, since there’s no DNS above (adding automatic updates to Route53 is something I have yet to bother with) how to get these into a load balancer without using ELB which can update dynamically with Auto Scaling. Well, using PuppetDB like I also previously wrote about.

A sample init.pp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class loadbalancer {
   $myhosts = query_nodes('hostname~"prd-web-', ipaddress)
 
   file {'/etc/haproxy/haproxy.cfg':
      ensure  => present,
      content => template('loadbalancer/haproxy.cfg.erb')
   }
 
   service {'haproxy':
      ensure    => running,
      subscribe => File['/etc/haproxy/haproxy.cfg']
   }
 
   package {'haproxy':
      ensure => latest
   }
}

Sample haproxy.cfg.erb

1
2
3
4
5
6
7
8

listen webstuff
   bind *:80
   mode http
   option httpchk GET / HTTP/1.1\r\nHost:\ www.acme.com
   appsession JSESSIONID len 32 timeout 3600
<% myhosts.each_with_index do |ip, i| %>
   server web<%= i %> <%= ip %>:80 check port 80 weight 1 maxconn 1500 inter 10000
<% end %>

And there it is, a rough construct of using Auto Scaling somewhat sanely with Puppet, and your own load balancer.

Because they’re too incompetent even to take peoples money.