James Jefferies and ShedCode Ltd

The world of James and ShedCode

Importing Network Rail Train Movement Feeds Into MongoDB

| Comments

Network Rail datafeeds?

Ah yes, please my previous blog post for further info, apologies for the lack of updates on here, I’ve been away, sent round the loop line.

Why MongoDB?

MongoDB is a scalable, high-performance, open source, NoSQL database. I thought it would make a good fit for the train movement feeds as it’s storage is oriented towards JSON-style documents. It’s JSON-style because it actually uses binary-encoded serialisation behind the scenes.

There is a nice little interactive tutorial on the MongoDB home page to get you in to the swing of things.

Getting Mongo installed on OS X

I use Homebrew for installing packaged stuff on to my Mac, if you use it, then you can easily install it with

brew install mongodb

What do you need for Ruby Mongo Magic?

There is a standard Mongo driver available, which you can install using gem

gem install mongo

but it is also worth installing the C compiled extension to speed up any BSON serialisation

gem install bson_ext

More info can be found on the official pages

Code example?

Here you go, proper version in the github repository

Examples of querying Mongo

There are lots of good examples and resources for using Mongo, but just a couple of examples related to the schema I’ve used in the example. You can fire up the interactive Mongo console and try the following (once you have some data!)

To retrieve all Departure events

db.td.find({ "body.event_type" : "DEPARTURE" })

To retrieve 10 (using limit) Train Movement (type 3) messages

db.td.find({ "header.msg_type" : "0003" }).limit(10)