James Jefferies and ShedCode Ltd

The world of James and ShedCode

Parse Train Describer CSV Data

| Comments

If you are using the Network Rail Open Data Feeds, then hopefully you’ve had a look at the fantastic wiki started by poggs.

In the reference data section, it provides a link to a CSV file which poggs is sharing containing Train Describer berths.

I wanted to parse that data using a simple ruby script and get it in to my mongo database.

You can use the mongoimport program to pull in the CSV by adding a header file, or you can use this little ruby script I wrote (with a lot of help from this snippet) which converts to JSON and inserts it in to the mongo db.

Before You Buy a Berg Little Printer…

| Comments

You may have heard today that BERG’s Little Printer is now available for pre-order. It does look like a great bit of kit & the ability to produce your own content should produce some interesting rolls of paper!

Anyways, before you splash out with your hard earned £199 + P&P you should, you really should have a look at Go free range’s Open Source printer.

Not only are they cheaper than the Little Printer, but being Open Source & built using Rails & Arduino, you can get stuck in to the source code, make changes yourself and create some interesting content.

As part of the Happenstance Project we were fortunate to borrow a couple of Go free range’s printers and start experimenting. We hooked them up with Twilio and a bit of ruby magic to get them recieving SMS messages. Each printer (Cathy & Heathcliff) had their own phone numbers, when they recieved the message, they printed it out & sent the contents to their twitter feeds.

You can read more on this blog elsewhere or you can watch our TEDx talk on the project as a whole.

So, buy all means pre-order a Little Printer, but have a think about getting the super duper Go Free Ranger printer first!

Cathy long cat

Importing Network Rail Train Movement Feeds Into MongoDB

| Comments

Network Rail datafeeds?

Ah yes, please my previous blog post for further info, apologies for the lack of updates on here, I’ve been away, sent round the loop line.

Why MongoDB?

MongoDB is a scalable, high-performance, open source, NoSQL database. I thought it would make a good fit for the train movement feeds as it’s storage is oriented towards JSON-style documents. It’s JSON-style because it actually uses binary-encoded serialisation behind the scenes.

There is a nice little interactive tutorial on the MongoDB home page to get you in to the swing of things.

Getting Mongo installed on OS X

I use Homebrew for installing packaged stuff on to my Mac, if you use it, then you can easily install it with

1
brew install mongodb

What do you need for Ruby Mongo Magic?

There is a standard Mongo driver available, which you can install using gem

1
gem install mongo

but it is also worth installing the C compiled extension to speed up any BSON serialisation

1
gem install bson_ext

More info can be found on the official pages

Code example?

Here you go, proper version in the github repository

Examples of querying Mongo

There are lots of good examples and resources for using Mongo, but just a couple of examples related to the schema I’ve used in the example. You can fire up the interactive Mongo console and try the following (once you have some data!)

To retrieve all Departure events

1
db.td.find({ "body.event_type" : "DEPARTURE" })

To retrieve 10 (using limit) Train Movement (type 3) messages

1
db.td.find({ "header.msg_type" : "0003" }).limit(10)

Moving From WordPress to Octopress

| Comments

What is Octopress?

Octopress is a blogging engine designed to be run from the command line, it uses Markdown for formatting content and doesn’t have any built in GUI editing facilities.

It is worth looking at though because it generates static HTML files, so you don’t need a database, you don’t need PHP you just need to serve straight forward files.

The integration with github pages is very nice, so you can actually use github to host your blog or website leaving them to worry about server admin etc!

Why did I switch?

At the moment, my virtual server was only hosting my blog and nothing else. At the moment, I don’t need it for anything else, so being able to move to github pages for free was one draw, saving me £X a month. When I need a server again, I’ll set one up.

I having nothing against WordPress per se, but it wasn’t very hackable by me, a non-PHP developer. Now with Octopress, it seems a lot more straight forward. Plus I no longer have a MySQL or PHP install to maintain.

It feels like the Octopress templates are more designer friendly. I’ll let you know if that is the case!

My site now has very straightforward version control, powered by git.

How does it work?

You check out the source (from github of course). The html generating engine is written in ruby, so you need to have that installed. Once set up and configured, you can push your content using git to wherever you want to deploy it.

When you deploy, octopress turns your markdown and HTML templates in to straight forward HTML pages ready to be served.

It uses Disqus by default for comments, but if you have a WordPress account, you can set that up with Disqus before migration. Works a treat.

Exiting WordPress

At first I tried the ruby WordPress migrating script. Basically it was not very good and I think I wasted a fair bit of time on it. Better is the Python exitwp.py script which is.. brilliant!

Downsides

  • There is a much smaller plugin selection available
  • You generally re-generate the whole site when you want to publish something new (there are ways around that though, to selectively generate new content).
  • If you had a blog with a lot of posts, then it might take some time to generate.
  • Small amount of off-the-shelf themes

Upsides

  • The standard plugins you might want are available, twitter, github, google analytics.
  • Github pages integration is very straightforward
  • Simple caching can be set up if required, but no more W3 Total Cache Super Plugin things
  • No WordPress security loopholes and patches
  • No PHP or MySQL loopholes or patches
  • Fast
  • Relatively lightweight page sizes
  • Nice default, responsive design

Theme

I’m using a slightly customised default theme at the moment, but hopefully will have a bespoke one up and running soon.

Summary

Now I’m migrated, I’m liking it so far, let me know what you think..

Downloading the Network Rail Schedules

| Comments

As well as the live feeds which you can subscribe to for the Network Rail data, you can also download the Schedule feed which is delivered by Amazon S3. The Schedule feed is an extract of train schedules from the Integration Train Planning System.

When you have your Network Rail account you can subscribe, either to all the data, or to the Train Operators you are interested in. However, at the time of writing, the only feed which works is the one for ALL the data, the Train Operator specific downloads throw an unhelpful error

HTTP Status 401 - User not subscribed to CIF_HL_FULL_DAILY

Which is a bit rubbish really. So my advice is, at this point to subscribe to ALL the data rather than just for one TOC. You can then access the feeds and download the zip file with either:

https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full

for the full feed or, say

https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-mon

for Monday’s update.

Note that you will need to authenticate if you haven’t already, using your username and password.

The Electrified Spine (and the Rest of the News)

| Comments

Today the government has announced a further set of infrastructure investment for the railways. This has to be good news, especially where electrification is concerned. Electric trains are more efficient, cleaner and perform better than their diesel (and steam I suppose!) counterparts. I appreciate that the electricity still has to come from somewhere, but in the main it is the right thing to do.

Already Announced

We know that plans to electrify the Great Western Mainline (Paddington to Cardiff, Bristol, Oxford & Newbury), plus the Trans-Pennine route between Manchester & York, Blackpool and Manchester to Liverpool are already underway. These are all good things, but there were some unsatisfactory omissions. The new announcement helps with some of the missing links, but not all of them.

There is a helpful map which shows these routes in glorious technicolour

New Plans - Midland Mainline

Hooray, the Midland Mainline, which has been electrified to Bedford for many years is to be electrified to Nottingham and Sheffield (via Derby). For those of us who regularly travel to London from Sheffield we should experience a faster service. Not necessarily more trains though, no extra track is to be laid and unless signalling is replaced, you can’t fit that many more trains on the track.

As far as I’m aware, there has been no announcement about rolling stock either, the simple solution for East Midlands Trains (assuming they’re still running the franchise) will be to get the existing Meridian trains upgraded with an extra coach containing all of the required electrical equipment, plus pantograph for current collection etc. Easier said than done, but a lot cheaper than brand new trains. So it is likely you’ll still be sitting on the same seat.

Missed Opportunities - Midland Mainline

Erewash Valley

Why or why isn’t the Erewash valley route included as part of the plans? This is the line which goes straight on at the junction for Derby and Nottingham, joining up with the Sheffield line between Clay Cross and Chesterfield - it’s the most direct route to Sheffield and is useful when engineering works close the route via Derby. Very few London bound trains use this route, most are routed via Derby from Sheffield, but electrifying it would be useful for diversions and could give the opportunity for Sheffield, Leicester, London services which don’t call at Derby, potentially shaving more time off the journey.

Sheffield to Leeds

Groan, everybody wants a better service between Leeds and Sheffield, the best being the 45 minute Cross Country packed like a sardine train which goes via Wakefield. Why not electrify that route too so that more London trains can be extended easily to Leeds? Plus get some more electrified suburban stoppers on that route. Derby to Leeds semi-fast?

Sheffield to Doncaster

Whilst we’re at it, how about the link to Doncaster too, providing more opportunities for diversions and commuter services?

New Plans - Oxford to Corby & Nuneaton

This wasn’t on my radar at all, but makes a lot of sense for new cross country routes as well as faster Birmingham to London trains via Oxford. Incorporating the start of the Oxford to Cambridge route (as far as Bedford) makes sense with that re-opening. The potential for new routes, well re-establishing old routes should give that part of the country a much better service.

New Plans - Reading to Southampton

Cross country trains mainly use this route at the moment and it is partially already electrified using the Southern standard 3rd Rail system. Over the next… err.. 50 years (?) the 3rd rail system will be phased out and replaced with the standard 25kv AC overhead system. This is the beginning of that process and also good preparation for Cross Country. Birmingham to Southampton trains will be electric all the way, as might, say Sheffield to Southampton trains, routed via Oxford.

The Spine

According to the plan, the Electrified Spine is to be the Southampton to Coventry and Sheffield routes and is a piece of spin. If I had a spine looking like that I’d be worried! The only new route, as far as I can see is the already announced re-opening of the Oxford to Bedford route, everywhere else is straightforward electrification.

Summary

I’m very happy to see these extra plans announced, but would like to understand more about why some very obvious links and connections have been omitted. What to others think?

Getting Started With Network Rail’s Datafeeds

| Comments

I’ve had a few railway project ideas knocking around for ages now, but have had a couple of blockers which have meant they have not progressed. Mainly lack of time and lack of data! I’m therefore clearing the decks for July and August to spend some decent time working on these ideas, which also co-incides with Network Rail giving access to a new platform providing live data feeds. You may have heard about the launch, I believe the platform was developed in conjunction with Rockshore, well, it’s what this tweet says anyway.

Where do you start?

If you are interested in using the datafeeds yourself, you can read all about it on Network Rail’s site which also has a link to download the developer’s guide in PDF format.

First, you need to create an account - I believe there are limited numbers of accounts going at the moment, so the sooner you do that, the better.

There are two types of feeds available, pub/sub using the stomp protocol for real-ish time data & downloads of huge files from Amazon S3 for timetables etc.

Once your account is active, you can subscribe to any of the pub/sub datafeeds you like and away you go.

Receiving Messages

I decided that I was going to try accessing the pub/sub datafeeds using Ruby (not my usual language) and put my experimental code on to my git hub account. Feel free to have a look and check it out, I’ll be using it for my initial experiments, so it should get updated beyond this basic example!

I’ve also included a sample, no error checking, very straightforward listener below. If you wish to try and run it, then you will of course need Ruby and Ruby gems installed, plus the stomp ruby gem.

Then, you just need to ensure your username and password are set as the relevant environment variables and away you go.

All being well, you should slowly receive updates from the topic you have selected. The example below uses Train positioning data for the East Midlands (TD_MC_EM_SIG_AREA) and assumes that you have subscribed to that feed using the network rail control panel

The updates you receive from this program are not formatted at all, it’s just sending the message straight to string. Of course, this is just helping you know you’re set up correctly. So you would get something like:

1
<Stomp::Message headers={"message-id"=>"ID:blahblah", "destination"=>"/topic/TD_MC_EM_SIG_AREA", "timestamp"=>"1341436026840", "expires"=>"1341436326840", "persistent"=>"true", "priority"=>"4"} body='[{"CA_MSG":{"to":"1234","time":"1341435963000","area_id":"WH","msg_type":"CA","from":"5678","descr":"1Z99"}}]' command='MESSAGE' >

Ultimately of course, you’d process the JSON from the body and do something with it.

Example Ruby Listener

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
require 'rubygems'
require 'stomp'

begin
  # Credentials set here as environment variables
  @user = ENV["DATAFEEDS_USER"];
  @password = ENV["DATAFEEDS_PASSWORD"]
  @host = "datafeeds.networkrail.co.uk"
  @port = 61618

  # Example destination add yours here
  @destination = "/topic/TD_MC_EM_SIG_AREA"

  puts "Connecting to datafeeds as #{@user} using stomp protocol stomp://#{@host}:#{@port}\n"
  @connection = Stomp::Connection.open @user, @password, @host, @port, true
  @connection.subscribe @destination

  while true
    @msg = @connection.receive
    puts @msg
  end
  @connection.disconnect
rescue
end

Week 10 - the End of the Happenstance

| Comments

At our Open House yesterday we had an ‘In conversation’ kind of thing with our mentor James Boardwell and what we thought would be a chat for 30 minutes or so, ended up being nearer to 2 hours! I knew we could talk about the last 10 weeks of Happenstance, that we had ideas and theories about what happened and why, but I hadn’t anticipated there being so much to talk about… in front of an audience!

When I went freelance last summer and set up my company, ShedCode, I was planning on doing some client work and in between, hopefully working on some of my own ideas to see what would happen. In the end I did 99% client work and was planning, at some point, to pause and work on my stuff.

That’s when Happenstance came along and it is fair to say that I thought at the beginning that Happenstance may have been a pleasant and interesting detour for my career as a Software Engineer/Developer and Techie. Instead of returning back to the path I was treading before Happenstance, I’ve finished the project on what feels like a completely different path, going who knows where!

Helping an arts organisation learn new things, to be inspiring and to fix stuff is what I thought would happen. We did some of that, but also I learnt what a Technologist could be, that we could make things, real things with lives of their own which will live on after the project has finished. I’ve met new people, made friends and found out about interesting things. An interest in electronics has been rekindled, thanks to arduinos and I’ve started to pick up some Ruby and a bit of Rails. Speaking at Future Everything, TedX Sheffield and at the Open Houses has given me some confidence in public speaking.

People have been so encouraging! Thanks to all who have had kind words to say, shared their knowledge, wisdom, contacts, experience, advice, tea and biscuits. Thanks too to those at Site who have been incredibly open, helpful and friendly during the project, I’ve really appreciated it. Special thanks of course go to my co-resident, Leila, who has put up with some of my more, er, random ideas, given me confidence to do stuff, been a great sounding board, co-speaker and partner in crime. She’s been so generous to me and I think we’ve been a great team (even if I say so myself!). I hope we can work together on more stuff soon!

I wrote a couple of blog posts about what to do next, both for me in general and technically and really it’s a work in progress. We’ve got debrief next week, then I’m on holiday for a couple of weeks. I think I really need a break, but I don’t want to lose the momentum I’ve built up during Happenstance. So, those who’ve become my friends over the last 10 weeks and those who’ve known me for ages, don’t let me slip in to old habits and ways & please bear with me as I bounce ideas around like fleas on space hoppers. I hope that life post-Happenstance is going to get even more interesting!

Whatever Next - the Technical Stuff

| Comments

In recent years my main programming language has been Java, sprinkled with a bit of the usual korn/zsh scrips, SQL, HTML, JavaScript and other gubbins. The world of Java is rich with libraries, web frameworks, platforms and alternate languages. I’ve invested a lot of time in this ecosystem, knowing which tools are useful for which problems, gained expertise in various web frameworks, development environments and platforms, generally to be as productive as I’ve ever been.

For the last couple of years though, I’ve been pondering whether to stick with what I know, or to spend time learning some new stuff. It is fair to say that I’m not a natural polyglot. I like to spend decent time getting to know a language and it’s ecosystem and I find it can be tricky switching.

Grails & Groovy

At first I spent a bit of time, starting to get to know Grails, basically using Groovy on the JVM (Java Virtual Machine) to build a Rails like web framework. The rest of the Grails stack is also stuff which I know very well, Spring, Hibernate etc. There is a lot I like about Grails and I’ve yet to investigate all this is new in the latest 2.0 release.

So, there you go, something I’d like, building upon a lot which I already know. Should be the favourite right?

Scala & Play/Lift

Well, I then started thinking a bit about Scala, a language which polarises opinion, especially amongst the Java community. It is either the ‘new’ Java or it is a hodge podge of static typed semi functional programming. People complain about it’s complexity or love how doing things in a more functional fashion reduces code and maintenance and gets it all set up for multi-core CPUs. If I was planning on doing more ‘corporate’ work, working on huge projects in London or building a twitter competitor then it’d make sense for me to have a good look at Scala.

But I’m not, maybe ShedCode will work on an idea which will one day be huge, I think I’ll worry about that another time!

Clojure

I’d also like to learn Clojure, mainly because I think it will be good for my programming skills and good for my brain.

Rails & Ruby

Then, working on Happenstance has made me think again…

In early 2008 I was going to investigate Rails & Ruby, one because I liked the fact it was called Rails, but then also because it seemed to be gaining a lot of momentum and interest and it was ‘cool’. In the end I got a job doing java, so I put it on the back burner. I also started to realise that, as is often the case where things are a bit ‘cool’ there were quite a few.. ahem.. nobbers around who not only believed that Rails was THE framework to use but that using anything else was such a POOR substitute. I have to be honest, it did my nut. I did meet a few people who seemed ok, but the majority, I didn’t really get on with.

Now I find that a lot of the R&D web apps I’m interested in, have been built with Rails. I’ve also met some more Rails devs who are not nobbers, in fact, they are interesting, helpful, open minded people. I’d like to work with some of them, I’d like to build things on top of the work they’ve done. I’d like to explore the Rails ecosystem which has inspired so much in other languages (like Grails for example).

Client Side

Even though over the last few weeks, I’ve been thinking more about whether the classic server side MVC web framework is what I should be looking at all. There is a significant trend for using client side JavaScript frameworks for doing a lot of the hardwork in the browser, whilst having a more simple RESTful service/API set up on the server. Examples would be Backbone.js, Ember.js, Knockout.js etc. This somehow feels like a good idea, but I’m not great with JavaScript and I’m not sure this is what I want to do just yet. Obviously improving my JavaScript knowledge would be useful regardless!

Crossroads

So, I find myself at a crossroads. I’ll spend some time with Clojure, regardless of what else I do, but I’ve got some interesting projects to start in the next month or so and I need to decide whether I:

  1. Learn Rails and see what happens…

  2. Go with what I know a bit and build something with Grails. I know the tools, I know the libraries and I know it works (mainly)

  3. Client side JavaScript stuff

  4. Go with Scala, build on top of the Java I know and become have decent with functional languages

At the moment, these are in my preferred order, but I’ll let you know what happens…

Week 7, 8 & 9 - Summary

| Comments

Oh dear, what happened to week 7, 8 blog posts (I hear you ask)?

Good question. Here’s a little re-cap of what you missed (thanks offbott for the reminders!)

Week 7

This week was mainly taken up with:

  • creating a test version of the site gallery website, so that we could have a good tinker with the layout

  • investigating the building’s network wiring and working out how we could get the office in to a better shape

  • an official rehearsal for our TedX Sheffield talk

Week 8 - the broken back week

I really kernackered my back on Wednesday morning, which meant that I ended up working a bit on .. ahem.. Cathy & Heathcliff’s back ends from home. Thursday was a bit better and we came up with the idea of building a ‘who’s in the house’ lighting board. Leila has been doing lots of soldering. Our IP camera arrived on Friday and we had it looking around the office with it’s spooky eye.

Week 9 - the TedX and Sheffield Doc/Fest Week

Tuesday was Sheffield TedX and we were invited to do a 10 minute talk as part of the whole shebang. We managed to get Heathcliff working by borrowing a MacMini from Site, getting it to pick up the Wi-fi in the Crucible (yes, THAT crucible in Sheffield) and share it’s ethernet connection. Attendees could then send SMS to Heathcliff and up in the refreshment area, he’s print out their messages.

We weren’t in the main part of the Crucible, but the Crucible studio which has room for about 270 people. I will confess and say that I felt the pressure for this one. Speaking in front of 270 people is one thing, being filmed ready for the TedX channel on YouTube is something else again!

I think we did ok and people had nice things to say about us. Hurrah! The other speakers were, in the main, fantastic! Inspiring stuff.

Then, Wednesday was the XO Summit as part of Doc/Fest, which I managed to… errr, attend. Bumped in to and had a chat with the Right Honourable Honor Harger and caught up with some people I’d met the day before.

Thursday was more Cathy & Heathcliff re-writing. Getting rid of the awful PHP which we’d used initially and the gmail to ifttt to twitter thing and replacing it with a bit of Ruby and Sinatra which I got running on Heroku (was it any co-incidence that Heroku had a major outage overnight afterwards? I hope so!)

Then today.. blogging ahoy!