Posts Tagged ‘testing’
Over the last couple of months we’ve started to use Canoo Webtest to functionally test our mobile web sites. We now use it for both product deployment cycles (we spin up a full version of bemokoLive and test a sample site prior to packaging up the platform) and site development.
What has particular impressed us about Canoo is the way we can rapidly create clean and succinct tests that address all of the functional aspect of a mobile web site that we can need to automatically test. There is a rich set of functions in Canoo to click around a site, interact with a page and test functionality and content of a page. We write all our scripts in Groovy which provides a language that is light on scaffolding syntax (i.e. code just what you want to do), allows us to dig into misc Java libraries as we choose and provides a functional language that is human readable (well to us techies).
For example, take the following test script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | package com.bemoko.webtest.live import com.bemoko.commons.test.webtest.BemokoWebtest import com.bemoko.commons.test.webtest.WebtestXmlParser import com.bemoko.live.devices.data.conf.HeroDevices import com.canoo.webtest.engine.StepFailedException /** * Test that welcome pages are rendered correctly to the hero devices */ class PagesTest extends BemokoWebtest { void test() { def parser=new WebtestXmlParser() // (1) Iterate over HeroDevices new HeroDevices().all.each { deviceData -> webtest("PageTests : ${profile} : ${deviceData.id}") { config(liveConfig) { deviceData.evidence.each() { header(name:it.key, value:it.value) } } // (2) Invoke the welcome page and verify some of the content invoke "/welcome/i", description: "Index page : ${profile} : ${deviceData.id}" verifyTitle "bemokoLive - i" verifyLinkedContent xpath:"/html/head/link[@rel='stylesheet']/@href", accept:"text/css" verifyLinkedContent xpath:"//img/@src | //input[@type='image']/@src", accept:"image/gif;image/png;image/jpeg" // (3) Validate document using an XML parser groovy { parser.parseText(step.context.currentResponse .webResponse.contentAsString) } } } } } |
The bemokoLive test suite provides a collections of devices that allow you to easily simulate requests from particular devices. The HeroDevices class, in the script above, provides a collection of around 10 devices that provide a representative coverage of different types of devices. We (1) iterate over this collection and pass in the evidence for the device, i.e. the HTTP headers, into the webtest configuration. We then (2) invoke a particular page, test its title and verify that the linked content is good. Finally (3) we validate the document against the DTD in the document – great for making sure nothing bad has slipped into the page.
Reports get generated with a full break down on the tests and any step failures along with a quick summary, for example the following screen shot shows the summary of test run of 431 tests which we ran against one of our customer sites. I’ve deliberately included a report with failures to make the summary a little more interesting:
There’s much more you can do with canoo and the Groovy scripting approach – take a read of automated testing on our bemoko wiki for some more pointers.
With this approach we can build up a robust set of scripts that verify our deliver and provide great foundations for our real device testing – no more burning time on real device testing dealing with functional issues that could have been caught earlier in development. This is also good black box test, so whether you’re using bemokoLive or a.n.other approach it’s equally applicable. If you’ve got a mobile web site that you want us to test, or help you get started to run this testing yourself just get in touch.
Judging by the size of the recent thread on the momolondon mailing list, the thorny issue of testing mobile websites is a hot topic.
There are many different software solutions around which will allow you to deliver markup to the myriad of devices currently available. Development for traditional, HTML based, websites has always been made more difficult as the different browsers interpret the markup in different ways and there are many tweaks needed to get the site to display correctly on all browsers. Interpretation of the standards, particularly CSS, seems to vary immensely (especially when some people, large companies based in Redmond for instance, seem to invent their own standards!).
This problem is multiplied many times over for mobile site development. Device independent rendering software solves a lot of the issues of screen size, memory limitations, paging etc – but what appears to be correct markup renders very differently on different devices. In my previous job, at a company that made device independent software, we had to make a large number of specific modules for certain devices to get round some of the stranger problems, one example that stands out is a device that would only display a whitespace after an anchor if you put three nbsp’s followed by a space.
Every device (even from the same manufacturer) seems to have different quirks displaying the markup but most of the problems occur in the different CSS implementations. Manufacturers either intentionally leave bits of the spec out or implement it incorrectly.
So, once you have your markup correct you have to start testing on a large number of devices to ensure your new big idea is displayed correctly. What’s the best way forward? There are several options:
- Buy every device and test each one
- Use emulators
- Use a service such as DeviceAnywhere
- Find a cheap way to hire lots of people to test on their own phones
- Automate the testing process
- Ignore it and hope everything is OK
Clearly buying every device is not an option given the number that exist and the rate of change. Normally devices are broken down into 5 or so different categories based on capability which gives a smaller sample size, but this can still be a large number of devices.
Emulators have their uses, but are not adequate for testing. Most of the emulators I have used are not accurate enough to test on and I have seen some that let markup through that crashes the real device.
Services such as DeviceAnywhere provide the actual devices to test on through a web interface, this is the best way of getting to a large number of devices but it is still time consuming to test on them all, and it can get expensive. There is also limited coverage at the moment so you can only test for a limited number of geographies. For a good overview of all the services available hop over to an article by an ex-colleague of mine, Tarek
If you know a lot of students you might be able to get some good testing coverage, but this would be tough to organise. A new service could address this problem, Mob4hire is like a social testing network which could be useful. There are also a number of options for this in the Far East, but network coverage could be an issue.
The holy grail here would be automating the testing process, but this is certainly a far off, if not impossible dream. You can automate testing the markup and this is useful for regression testing but you cannot automate looking at the display on the phone to check it.
If you think ignoring the problem and hoping it will be OK is the way forward, then you are probably in the wrong industry.
I would like to have been more positive in my first blog for bemoko and to reward you for getting this far with the answer to testing on mobile devices. Unfortunately I can’t provide that answer but bemoko has a lot of experience in testing sites that have been produced and, while it’s not perfect, a good strategy can be built to give a decent testing coverage using a mixture of the methods I have outlined above.