After being Technologist in Residence at Site Gallery earlier in the year, as part of the Happenstance Project I started to think of some ideas I wanted to follow for later in 2012.

Hackday

One of those ideas was to run a Hackday, but not any old Hackday, one centred around Railways. Network Rail had just opened up their realtime Datafeeds and as far as I knew, no one else had ever run one. Sure, there had been Transport Hackdays, but not Railway specific ones.

First steps

So I got to work, I went to experience a 24 hour hackday for myself, where I discovered that sometimes technical issues with the building can really slow people’s progress and that working overnight doesn’t really suit me anymore!

Clickerty Hack

I also bought a domain and registered a Twitter account. The hackday would be called Clickerty Hack. Then, having a good read of The Hackday Manifesto made me realise that a lot of work was required to get a decent Hackday up and running.

Loco 2 and Off The Rails

It was then that I found out about somebody else’s plans for a Railway Hackday, loco2 was a company I was already watching with interest. A startup based around European Train travel, ticketing and guides.

Their hackday, Off the Rails, was in a more advanced state than Clickerty Hack, so putting it on hold, I volunteered to help.

ShedCode Sponsorship

It was a real privilege to be able to help with a small part of the organising and for my company ShedCode to be one of the sponsors, providing some of the food for the masses!

Meeting the experts

During 2012, I’d found a number of people who had built some incredible pieces of software using Railway data, especially Peter Hicks and Tom Cairns who built, amongst other things Open Train Times and Train Times respectively. So it was brilliant to meet them in person at Off The Rails. There were also many other people knowledgeable about Railways, Open Data and Mapping - I could have spent all day just talking to people!

The teams

During the day, the teams built some inspiring things, including an application which builds spotify playlists to match the length of your train journey and many others. Most of them will hopefully be available at some point on github for perusal.

Thanks

All in all, a fantastic day, thanks to Loco2 and Bethnal Green Ventures and here’s to the next Railway Hackday, whether it’s Off The Rails 2, or Clickerty Hack! With cake too of course ;)

Corporate Social Responsibility

Corporate Social Responsibility is a huge field in modern times. Businesses vary in their response to the challenge of responsibility, from box ticking a checklist, to doubling up charitable donations, to actively encouraging employees to be involved in Charitable works.

My old company, Technophobia ran a FedEx day, where one of the challenges was for a team to rebuild a website for a charity, which they did. You can read all about it, good for them!

This year though, I’ve been thinking more about how companies could donate their employees and employee time to make a difference in more unusual ways.

Technologist in Residence

Having had the privilege of being a Technologist in Residence at the Site Gallery earlier this year, I was able to see first hand what a difference a problem solving techie can make in a relatively small art’s organisation. With a Research and Development remit, as well as helping with some of the day-to-day tasks, a techie can enjoy the excitment of trying new things, whilst also being able to fix problems without the usual gubbins of working for a large corporation.

Of course, many arts organisations have a resident techie, but usually they have to spin many plates, from producing exhibitions, to maintaining the website, leaving little room for them to catch up with some of the more strategic things which need doing.

There is limited funding to run Happenstance style residencies, and this is where local businesses could make a difference. By adopting a local arts organisation, getting involved, running events and allowing staff to spend a week once a year, then a day a month for the rest of the time, for example, the benefits would be felt by all.

Who gains?

• The arts organisation - they gain some help in fixing some things, as well as being inspired with new and interesting technology. Building relationships with the local tech community
• The techies - they can hopefully have time to do experiments, R&D, as well as doing some of the stuff which needs doing. From tidying networks, configuring firewalls, mending printers, redesigning websites, helping with copy… you get the picture!
• The company - they help the local community, building relationships with organisations around and about, increasing their reach and network. Their employees are more inspired and excited about new things.

Who pays in these difficult times

It would be completely understandable for tech companies to claim that they cannot afford to let their billable employees take time out of their billing cycle to do ‘good’. Understandable, but short sighted. It’s the same argument about sending employees on courses, or to conferences - “where is the business benefit?”.

Well, I believe that good, motivated employees are expensive to recruit and maintain, why risk them getting demotivated, bored and unhappy, when by giving them time to innovate and collaborate, they grow! So the risk is far greater, that you lose good people by not giving them the room to grow. Best to pay out of a sense of innovation and employee investment, than to pay the price for losing them, either by them leaving or by turning them in to faceless, billable resources.

Following on from the amazing Happenstance Project I worked on earlier in the year, we had the final Showcase at Nesta HQ. For the Showcase, Leila produced a Happenstance Newspaper, designed by Alex Parrott including articles by yours truly! You can download a PDF copy of it.

Enjoy!

The Creativity Tap

Creativity is just like a tap right? You need to do some thing a bit woo and a bit wah, so you switch on the creativity tap and all these incredible ideas just pour out of your brain. There are two dangers though if you use the creativity tap.

One is that if you leave that tap switched off for a while, then all the creativity gets stored up in your brain, without an outlet until… Pop! One day it’s gone, those ideas had to go somewhere, so they leaked out, usually somewhere behind your ear, hopefully picked up by someone else. You have writers block, coders deadlock, songwriters… err, song block. Oh dear

The other danger is that you leave that tap switched on all the time until your brain empties. This too is a bad idea, sure, you’ve made some room in your noddle for new things, but you need a few left in there, to keep the ideas factory ticking over. Oh double dear…

Writing Songs

Knowing people who are songwriters, I get the privilege of seeing songs appear, until they end up being recorded and performed, or they mutate into a different song, or they end up on the shelf, maybe for another day. I’m sure not every songwriter works like this, but those I know have to work at honing a song. They rarely appear fully formed but they often appear when there is space in their lives to switch on the creativity tap for a bit to see what happens.

If they don’t have the chance to get the songs out, into the open, then you can see how those song ideas start to play merry hell. The frustration builds, no time for the songs, too busy having to do other stuff. It’s almost a physical pain, the song has to be brought to life. There is a creative imperative which must be fulfilled.

Writing Code

Fortunately, for us software engineers, coders, developers, tapping at our keyboards day by day, telling the computer what to do and when to do it, we don’t need the creativity tap. We get stuck? We ask a search engine. We need to do something from scratch? Well, someone will have done something like it, we just need to find it and copy them. We need another pair of eyes? Well, ask a colleague. It’s all there for us, if we use a clever framework or development environment, it’ll probably autocomplete what we’re thinking too. We are coding machines..

Or are we? Having spent many years writing code, I started to realise that when I was mechanistic in what I was doing, I became frustrated that I wasn’t able to come up with new ideas or ways of doing things. I was constrained by getting this piece of work finished in as short a period of time that was possible. No time to switch on the creativity tap, just get it done.

The big problem is when you work for someone else, you can find yourself under a lot of pressure to crank the sausage machine handle rather than think things through. I worked with an engineer once, who would deliberately stop what he was doing and just think. Rather than thrashing through a jungle of code… stop first and think. It worked too, giving a bit of time to turn on the creativity tap to see what happened.

I struggled in my day to day job, so I tried to make time in the evenings and the weekends. But you’re so tired then that switching on the creativity tap just floods you with ideas which you don’t have the energy to either make note of or do anything with. Like the frustrated songwriter I would feel the pain of not learning new skills, not playing with languages and technology. Not actually chuffing making anything which made you say, even if it is just to yourself “I made that and I’m pleased with what I’ve done”.

So, what can be done?

Well one answer, in the workplace, where people have a pressurised, day to day job and a boss, is for the bosses to give people time when they have the headspace to play. With their work, with something new, with wooden blocks, with a great idea they’ve had. Not only do they get chance to switch on the creativity tap, but usually it knocks on in to their ‘day to day’ work too. Wins all around.

If you are your own boss, then plan in time where you put down the work you’re doing for clients or your startup and have a play. You never know what amazing ideas might come to life!

Disclaimer - the creativity tap premise is a load of nonsense, if only it was a tap we could switch on and off!

The problem

After migrating a WordPress site, auto update and plugin install wouldn’t work. The former errored with could not connect to host which seemed very odd.

First attempts

WordPress allows you to download the file manually or autoupdate if configured correctly. Clicking on the manual download link and the file downloaded fine to the local machine. So first, I thought I’d try and download the file from the command line on the new server.

1

wget http://wordpress.org/wordpress-3.4.1.zip

Zip, it all downloaded fine.

Extra help

There is a good trouble shooting/dev plugin for WordPress called Core Control. By installing this and enabling the HTTP control, you can test the various transports WordPress uses for downloads. Curl, PHP File open, PHP Fsockopen etc.

Testing all three default transports, starting with cURL came up with the same message, could not connect to host

Head Scratching time

So WordPress was telling the truth. Now, as WordPress was running as the www-data user, what happens if I tried the wget from the command line for that user?

1
2

wget http://wordpress.org/wordpress-3.4.1.zip
Permission denied!

Ah-ha, so it was nothing to do with WordPress, the www-data user couldn’t download anything.

Problem solved!

Firewalls, I started thinking firewalls and lo and behold, BytemarkUK block outgoing http traffic for the www-data user by default! This makes sense for when sites get compromised, but it also stops some of the core functionality for updates etc working.

Enabling http traffic for the www-data user will make things worse if the site is compromised, but also makes it a lot easier to keep the site up to date, reducing the chances of a WordPress exploit.

The following removes the firewall rule

1

rm /etc/symbiosis/firewall/outgoing.d/50-www-data

There seems to be a lot of noise at the moment about the latest security vulnerabilities in Java, especially Java 7. I’d go as far as to say that some of the articles are scare stories. Is this what we need, to give Java the boot?

TL;DR

All computers connected to things have security vulnerabilities of various forms.

Desktop Java installed computers where not required is an unnecessary security risk and should be uninstalled, just as Adobe Flash, Adobe Reader, Silverlight etc. Windows 8 and Mac OS X don’t bundle Java runtimes via default anymore. This is a good thing. Oracle should be far quicker out of the blocks fixing security vulnerabilitles.

Giving Java the boot full stop is an over the top reaction. However, giving Java the boot on the desktop when it’s not required is a wise move.

History

Java as a platform has a reputation for being secure, reliable and frequently patched. Being able to run Java server side on Sun software provided the building blocks for many web businesses at the end of the last millenium and in to this one.

It also has many branches in to other parts of the internet ecosystem. As well as running server side, the other two main areas are mobile Java, J2ME, for example and desktop Java, either via browser plugin Applets or as runnable GUI applications using a desktop JVM (virtual machine).

Over the last few years, mobile Java has basically been nobbled by the rise of smartphones and their respective application frameworks and stacks, Android, iOS, Windows whatever it’s called, BlackBerry and the rest.

However, the legacy of desktop Java has continued by some applications still requiring a JVM installed on the local PC and some people still writing or using Java applets. Althoguh Sun and now Oracle have continued to try and encourage desktop development with their JavaFX initative, they’ve not got very far.

Applets

The idea behind applets was that you visited a site via a web browser, which could always deliver the latest version of the application code to your browser, either to run in a plug-in or using the JVM installed on your machine. They were useful for some niche applications, you sometimes see them now replacing flash file uploaders for example, but the User Experience was often a bit shonky.

Desktop Applications

Write Once, run anywhere was the mantra behind Java Desktop applications, an attractive proposition. One set of code would run on Windows, Linux, Unix variants & Macs. Many Java development environments have been written in Java and are used today. CrashPlan is an example cross platform application which requires desktop Java to work.

Legacy

Unfortunately, a lot of the Java runtimes (and Flash installs, adobe reader installs etc), installed in days of yore on people’s machines will have security vulnerabilities, either because they are out of date or because of recent new exploits.

It was interesting to see Apple’s response to recent JVM exploits, although they may have been slow in responding, they, in effect, switched off people’s Java installations, providing a prompt if it was required. In one OS X update (assuming it was installed!) Macs became better protected from Java exploits.

Summary

If you don’t use Java on your machine - uninstall it, it’s not worth having a potential vulnerability. Same with Flash or Adobe reader - if you don’t use them, uninstall them.

If you do need it installed on your machine, there is some great advice here

Of course, keep things up to date that you do use, Operating systems, browsers etc

Java server side is a different story. If you do use the Java Virtual Machine, whether it is for Java apps, Clojure, Scala or whatever. Keep your eyes on the Oracle security bulletins, update when you can and be aware of any vulnerabilities in the latest version.

Back, back, back in the day… well, January 1978, Kate Bush released the classic song Wuthering Heights, inspired by the Emily Bronte novel. As well as launching her career, and boosting the sales of flowy white dresses, the song provided inspiration for us when it came to one of our projects.

Go Free Range

James Adam of GoFreeRange has done a lot of work on transforming simple thermal receipt printers in to intelligent, ethernet enabled printing devices. He’s done an incredible job of writing software and working out the electronics, based on the Arduino device, to get them up and running.

Cathy & Heathcliff

When our two printers arrived to play their part in the Happenstance project, their default names were a bit of a mouthful, they made a good password, but not a name which slipped off the tongue - 4j8k6i5x5q7a7b5y.

Up and running

So, when we finally got the printers up and running, after a few head scratching hours, the first message we sent to 4j8k6i5x5q7a7b5y was, ‘It’s me, Cathy, I’ve come home’ – a line from Cathy to Heathcliff in the song. So Cathy named herself, and 4j8k6i5x5q7a7b5y became Heathcliff.

That was in the final week of the Bill Drummond exhibition, “Ragworts” at Site Gallery and we gave the two printers a part to play in the project. Some local Sixth Form students were tasked with sending messages to Cathy & Heathcliff as they jumped on the nearest bus they could find, and journeyed into the wilds of outer Sheffield.

At first we thought they could tweet their progress and the printers would pick up on those tweets and print them, but we decided to mash up a lot of technology to provide a simpler solution for the students. With the clock ticking, we glued together Twilio, Gmail, ifttt (‘If This Then That’) and some custom code we wrote. Cathy & Heathcliff now had their own Sheffield phone numbers, and now you could send them a text message and the contents would be printed and tweeted.

When the students set off on their journeys, they could send SMS text messages to Cathy and Heathcliff’s numbers and the magic of technology would take care of the rest. Their messages would appear on the printers, back at Site Gallery and on their Twitter account. It was fascinating to read the messages sent back to base by the students, many of which were very poetic and evocative.

After all this excitement, our plan was to get Cathy and Heathcliff mobile so they could roam the streets of Sheffield whilst remaining available for printing. With the aid of dongle, a mac and a giffgaff sim card they were freed from the shackles of mains power and wifi, ready to explore!

What’s next

Although Happenstance has now finished, the printers have been used in Site for other purposes. Cathy has been on reception acting as a comments book (text her and she’ll print your comment) whilst Heathcliff has been reading twitter and printing mentions of Site Gallery!

We still hope to use them for one final experiment where we get them to run off steam power!

So, your server is set up and raring to go. Please see my earlier blogpost on getting the server to this point.. next!

Install Apache2

Let’s get our web server up and running by installing apache

1

apt-get install apache2

and check it is up and running with http://123.456.789.012 in your friendly web browser.

Configure Apache2 for low memory usage

With previous virtual servers, I’ve found that Apache needs a bit of tweaking to support WordPress load. I found this useful post about tweaking apache. The nuts and bolts are to amend some of the settings in your /etc/apache2/apache.conf as follows:

1
2
3
4
5
6
7
8
9
10
11
12

Timeout 30
KeepAlive On
MaxKeepAliveRequests 50
KeepAliveTimeout 10

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

Setting up virtual host with apache

A very simple set up would be to create the following /etc/apache2/sites-available/myacedomainname.com

1
2
3
4
5
6
7
8
9
10
11
12
13

#Ensure that Apache listens on port 80
#Listen 80 (note uncomment this with ports.conf set to 80 and you will experience a WORLD of pain!)

# Listen for virtual host requests on all IP addresses
NameVirtualHost *:80

<VirtualHost *:80>
DocumentRoot /var/www/myacedomainname.com
ServerName www.myacedomainname.com

# Other directives here

</VirtualHost>

Enable site (and other useful commands)

Create that file in sites-available and use the following commands to enable/disable - they will give you options when they run as to which sites you want to enable/disable

1
2

sudo a2ensite
sudo a2dissite

Force reload configuration?

1

sudo  /etc/init.d/apache2 force-reload

Install PHP

Ok, well to get WordPress running, we’re going to need some PHP, so get it installed

1

apt-get install php5 libapache2-mod-php5

Install MySQL

And we’ll need a database too

1

apt-get install mysql-server mysql-client php5-mysql

If already installed, but you want to reset it so you can set root password etc, give it a bit of this:

1

sudo dpkg-reconfigure mysql-server-5.1

Migrating a site from another provider

Database gubbins

Get a database dump from your old site, either using PHPMyAdmin if installed, or doing a dump from the command line using mysqldump. Hopefully at the end of that process you’ll be able to import the dump in to your new set up with a bit of:

1

mysql -uroot -p < databasedump.sql

The dump may contain a create database statement to create the database, right at the top along the lines of

1

CREATE DATABASE `db1234` DEFAULT CHARACTER SET latin1 COLLATE latin1_german2_ci; USE db1234;

Now you’ll need to create a database user for WordPress to use. So make sure you’ve got a file dump of your WordPress installation, either by command line, or the hosting admin suite, or an FTP application or whatever you’re comfortable with. You’re going to need your wp-config.php file which will tell you how your existing WordPress site is set up. This is the bit you need to look at for now

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

/** The name of the database for WordPress */
define('DB_NAME', 'db1234');

/** MySQL database username */
define('DB_USER', 'dbuser');

/** MySQL database password */
define('DB_PASSWORD', 'dbpassword');

/** MySQL hostname */
define('DB_HOST', 'localhost');

/** Database Charset to use in creating database tables. */
define('DB_CHARSET', 'utf8');

/** The Database Collate type. Don't change this if in doubt. */
define('DB_COLLATE', '');

So in your existing set up you have a database user called dbuser with password dbpassword so create that user in your new setup as follows

1

grant all on db1234.* to 'dbuser'@'localhost' identified by 'dbpassword';

Then make sure it works with a bit of mysql -udbuser -Ddb1234 -p enter your password and hopefully you are in.

Filesystem

Now get that dump of the filesystem you have and unpack it in /var/www/myacedomain.com or whatever you set in your virtual hosts configuration earlier. You may need to change the ownership of everything, including that directory to www-data with a bit of

1

sudo chown -R www-data:www-data /var/www/myacedomain.com

and check the permissions too, directories should be set to 755 and files to 644. You can enforce this with

1
2

sudo find . -type f -print | sudo xargs chmod 644  # Files
sudo find . -type d -print | sudo xargs chmod 755  # Directories

It is entirely possible that after all this, you can now hit http://myacedomain.com or http://123.456.789.012 and see your migrated WordPress site!

UTF-8 issues

Now, one issue I’ve had in the past is problems with UTF-8 conversion. I.e. the posts you’ve migrated end up with funny characters like – “ ’ †- you can tell if this is causing you grief because you see the characters mixed up in your posts, but if you comment out these lines (by adding // in front) in your WordPress config, then they.. ahem, magically go away

1
2

define('DB_CHARSET', 'utf8');
define('DB_COLLATE', '');

Best not to do that though, you can use the search and replace WordPress plugin to replace all the dodgy characters in the database. Find “â” and replace with “” - hooray!

Accessing the server

Once set up, you will have the IP address, in the form 123.456.789.012 and the root password. This means you will be able to log on to the server using ssh (secure shell), i.e.

1

ssh root@123.456.789.012

You will get a message along the lines of

1
2

The authenticity of host '123.456.789.012(123.456.789.012)' can't be established. RSA key fingerprint is blah blah bla
Are you sure you want to continue connecting (yes/no)?

to which you will reply

1

yes

and you will then get

1
2

Warning: Permanently added '123.456.789.012' (RSA) to the list of known hosts.
root@123.456.789.012's password: 

which is when you can enter your password. Hopefully then you’ll be logged on to the server.

1
2

blah blah blah snip…
servername:~# 

First things first

Root access is the super user for your server. You really don’t want to have people being able to log on to your server as root. What you really need is to have user accounts which people can use to log on, and if/when required, they can get super user privileges to do whatever they need to do.

What we are going to do is to stop root being able to secure shell straight in to the server, create a new user account to use for our admin (with upgradable privileges) and run ssh on a different port to deter basic attacks.

Change root’s password!

So you have a default root password - CHANGE IT to a new one! From a root prompt, use passwd to change the password

Adding a new user

1

adduser james

Will create a new user, called James - you can add in extra details as you go like full name etc.

Now we want to allow james a bit of privilege, not by default though, but by using the sudo command.

Aside - setting default editor to be vi

By default, for all users, you may want to use vi as your editor (ok, you may not, but I do!) so add the following to /etc/profile

1
2

EDITOR=vi
export EDITOR

Giving the new user sudo privileges

You may need to install sudo if it isn’t already installed on your server. It’s a simple case of

1

apt-get install sudo

Then, the application visudo is your friend. You edit the config file using visudo and then sudo does the rest. When you open the file, you’ll see a section with:

1
2
3
4

# User privilege specification
root    ALL=(ALL) ALL
admin ALL = (ALL) ALL
james ALL = (ALL) ALL

By adding my own entry, it allows me to upgrade my user to super user privileges for all actions. You can limit the commands which users can run if you like using this, but if you are going to be the super user, you probably want to leave it as all.

Sorting out access

Let’s stop people attempting to log on to the server as root via ssh and also to run ssh on a different port than the default. There is a lot of good advice here

Amend port

The default port is 22, change this to something else, which isn’t being used by anything else

1
2

# What ports, IPs and protocols we listen for
Port 60

Stop root login

Now stop people logging in as root

1

PermitRootLogin no

Some other bits

Give maximum number of log in attempts to be 3, only allow james to login.

1
2

MaxAuthTries 3
AllowUsers james 

Restart sshd

As root (or sudo)

1

/etc/init.d/ssh restart

Now try logging in remotely from a different shell (i.e. keep the one you’ve just restarted sshd on open in case you have any problems!)

1

ssh -p60 james@123.456.789.0123

obviously set the p to be the port you set earlier.

Stop chancers getting in

It’s also a good idea to stop people attempting to log on to your server using common passwords, usernames etc. A good way to do this is to install the fail2ban which goes some way to banning people trying to brute force their way in to your server.

1

apt-get install fail2ban

Default settings are pretty good here.

Firewall configuration

UFW or Uncomplicated Firewall is a good first port of call for securing your shiny new server. You can install it as root with

1

apt-get install ufw

Now, unlike fail2ban, ufw is installed switched off. You need to configure a few things before getting it up and running and providing that extra security. By default ALL ports are shut, so make sure you’ve opened up the ones you need before switching the firewall on!

For example, if you’ve set up your ssh to run on port 60, then you need to run

1

ufw allow 60

If you’re going to be running apache on port 80 (the usual)

1

ufw allow 80

Here are a few notes

Don’t forget to run ufw enable when you are ready! ufw status tells you what is set up.

Let us say that you have a MySQL dump of a database table, with a load of data which ideally, you’d like to pull into a Mongo DB as json.

What you could do is…

Import data in to MySQL database

Usual kind of thing here, assuming you have a MySQL username for a created database:

1

mysql -uusername -p -Ddbname < mysqldumpfile.sql

Export data from MySQL database in json format

There is a cracking ruby gem mysql2xxxx which will export from a mysql database to various formats, including json.

You can drive it via code or a number of binaries are provided. So:

1

mysql2json --user=username --password=password --database=dbname --execute="select * from my_special_table" > my_special_file.json

Importing the json straight in to your mongodb

Then you can use mongoimport to pull in the json. If your export from mysql2json is surrounded by the square brackets, you’ll need the --jsonArray option. Your command will need to be something like:

1

mongoimport --type json --file my_special_file.json --collection mycollection --db mymongodb --jsonArray

Summary

Simple three step transformation from MySQL dump to Mongo. You may know a better, more straightforward way of doing this, please add a comment if you do!