Entries in device lab (2)


Device Lab progress

A month or-so-ago we set about building a device lab to help with our cross-device testing. Once we'd figured out which device and operating system combinations we wanted to start with we looked at the cheapest ways of obtaining them. Some had to be bought new, but others could be picked up relatively cheaply 2nd hand.

Early December was like Christmas-come-early with a flurry of devices arriving at the office. Pretty much learning as we went, we started figuring out a set up process to help make things easier as new devices came in. We're still adding to it and most things are commonsense, but if you're thinking of setting up your own device lab then there may be some useful points:
  1. build an asset register. To help keep track of what devices we have, where they came from and associated peripherals it's useful to maintain a register.
  2. turn-off auto updates. Where possible this applies to both the operating system and browsers/applications. We purposefully picked up some devices that were running older versions of an OS and it was important to make sure we maintained these.
  3. provide WiFi access. We're planning on using a variety of tools to help make testing across multiple devices easier. These all rely on being on the same network. 
  4. install browsers and choose a default. On a number of the devices we've installed a mixture of browsers. Installing these upfront and choosing a default makes life easier – where possible remember to turn off auto update on browsers if you want to maintain a specific version. 
  5. clear the browser history and cache. If you've picked up a 2nd hand device then it'll be useful to clear the browser cache (this is also good practice for post-testing). Most of the devices we picked up were already wiped, but one went straight into gmail logged in as the previous owner, eep!
  6. update display settings. Whilst testing the chances are you won't be directly interacting with all the devices all the time. Updating the display settings so that the device doesn't automatically turn-off the display will save with the hassles of repeatedly unlocking or adjusting screen brightness.
With the devices set-up it was time to look at how best to test across multiple devices at once. We looked at a couple of options including Adobe's Edge Inspect. But for the time being we're moving forward with using Ghostlab – we found this worked across all the devices, had plenty of flexibility and handy debugging using Weinre. (Thanks to folks who commented on the previous post suggesting looking at these!).

Our set-up at the moment isn't perfect, but we're about to go through a major office refurbishment after which we're hoping to have a dedicated space for a permanent device bench. Once this happens the next step will be to get ourselves set-up such that we can open the device lab to local developers.

We currently have 14 devices with a variety of operating systems. The list at the moment looks like this:
  • ZTE Open – Firefox OS – Gecko 18
  • Samsung Galaxy SII – Android 4.1.2 (Jelly Bean) – Android 4.1.2 (WebKit 534)
  • Nokia Lumia 800 – Windows Phone 7.5 (v797.01) – IE9
  • Hudl (Tesco) – Hudl 1.3 (Android 4.2.2) – Chrome (28.0.1500.94)
  • Samsung Galaxy Tab 2 – Android 4.1.1 (Jelly Bean) – Android 4.1.2
  • Apple iPhone 5
  • Apple iPad 4 – iOS 6.1 – Mobile Safari
  • LG Nexus 5 – Android 4.4 (KitKat) – Chrome (v30.0.1.5999.105)
  • Kindle Fire – Android v2.3 (customised for Amazon Kindle) – Amazon Silk
  • BlackBerry Curve 9320 – BlackBerry OS 7.1 – Blackberry Browser
  • Nexus 10
  • Nokia Lumia 820 – Windows Phone 8 (8.0.10328.78) – Internet Explorer 10
  • Blackberry Z10 – Blackberry 10 ( – Evolution
  • Surface – Windows RT – Internet Explorer 10

Building a device lab

At the Money Advice Service we're currently working on a series of web improvement initiatives. This includes reviewing our existing taxonomy, site search strategy and also building a new responsive front end. For the past few months the tools we’ve developed in-house have been built in a responsive manner, however once live they’ve lived within an unresponsive frame. The enhancements from the web improvements initiative will change this, and along with our goal of being ‘mobile first’ we’ve been thinking about the tools we need to help design, develop and test across small screen devices.

One of the things we’re doing is building a physical device lab that’ll complement the use of online emulators. This is at a fairly early stage, but over the course of the next few weeks we'll be moving forward and I’ll update with progress. For now I thought it might be useful to write some notes around why we’re building a device lab, how we decided on the devices to start with, and what our aspirations are for the future.

There are a number of reasons why we’ve opted to invest in a physical device lab over relying purely on emulators. Emulators definitely have a place and will be a part of our workflow. They provide good coverage and can be quick and easy to use, especially on static, content based sites. However having access to the physical device provides a number of advantages:

1 – Firstly, it’ll be no great surprise that the number of visits to the site on small screen devices has increased rapidly. Looking at analytics data* there’s been an average increase of 398% year-on-year. The increased usage meant that optimising the site for small screen devices became a growing priority, and with that the need for more effective testing.

 2 – an emulator can replicate how a site will look given a device’s particular capabilities (screen resolution, browser support etc) but some of the subtilise around how the site will behaviour on a device can be lost, for example the responsiveness of a touch screen or having to navigate using a trackball or keys.

3 - the third advantage is that having physical devices available on site really re-enforces the mobile-first mindset right across the team, from product owners, to designers, to dev and QA. It means that everyone can look at the latest product developments across a variety of devices rather than just the thing that they have in their pocket or on their desk.

Getting the device lab set-up
Having decided that we wanted to a physical device lab the next question was what devices to get. Thankfully the interwebs came to the rescue and we were able to look at plenty of useful resources from the likes of Jeremy Keith, PPK and Stephanie Rieger. They’ve all written a bunch of stuff on the topic, and are much more knowledgable than me, but I figured I’d summarise a few of the things we found useful.

1 – in the first instance we looked at existing traffic to the site to see what the most common devices, OS, and browsers were within our existing audience. The data showed us that when it came to small screen devices the vast majority of traffic came via Apple iPad and Apple iPhone devices (36% and 33% respectively) with a further 7 devices making up 80% of our total mobile traffic. Looking the operating systems, 80% of our mobile traffic came from 12 devices, of which 9 were different variants of Apple's iOS. The browsers hitting the site also told a very similar story.

2 – looking at out own data analytics provided a great snapshot of our immediate audience, however we also wanted to consider the wider device/browser landscape. At the Money Advice Service our target audience is the whole of the UK and our goal is to make everything we produce available to as many people, in as many ways, as possible. Previous experience has shown that in making a site optimised for mobile then traffic from mobile devices is likely to increase and broaden. To help get a wider view we looked at data from GDS and data.gov.uk.

3 – with a view of device traffic to both our site and our target audience we were confronted with a fairly daunting list of potential devices. Through looking at existing device labs we were able to take some inspiration as to the best combination of devices for us to start out with. Combing this information from the data we’d captured from the first two points gave us a good starting point.

4 – finally, we looked at the devices we already had in the office. Having a set budget we wanted to be frugal with our spending. The most common devices we wanted to form part of a dedicate lab, but some of the less popular devices we didn’t want to duplicate what we already had in the office knowing they could probably be called on for testing purposes as and when needed.

Where we are now, and where we want to be
Right now we’re at the point of starting to collect devices. Newer ones have been ordered, and for some of the older models we’re scouring eBay and 2nd-hand electronic shops. By Christmas we should have the basis of a good device lab.

We’re also starting to figure out a way of making testing as easy as possible. We'll probably using Adobe Edge Inspect and Forward to help with this, but very keen to get any tips or suggestions! There’s also the logistics of how to manage devices, store them, and maintain the right version of operating system. As we figure some of this stuff out I plan to write a follow-up post on some lessons learnt.

Our aspiration for the future is to be able to open up our device lab so that others can also benefit from it too. The potential of being able to share access to devices and with that learn from each other about problems we’re facing and ways to overcome them is hugely exciting. We’re in the process of starting an office refurbishment that’s due to be completed in the spring of 2014, so hopefully that’ll give us the opportunity to set something up for then.   

* data taken from April 2010 through November 2013