LinkedIn Facebook Twitter bemoko blog bemoko technical pages
Tel: +44 1256 581028 Email: info@bemoko.com

Bemoko blog

26

Jan

Testing mobile sites

Posted By Mat Diss Saturday, January 26, 2008. 5 Comments

Judging by the size of the recent thread on the momolondon mailing list, the thorny issue of testing mobile websites is a hot topic.

There are many different software solutions around which will allow you to deliver markup to the myriad of devices currently available. Development for traditional, HTML based, websites has always been made more difficult as the different browsers interpret the markup in different ways and there are many tweaks needed to get the site to display correctly on all browsers. Interpretation of the standards, particularly CSS, seems to vary immensely (especially when some people, large companies based in Redmond for instance, seem to invent their own standards!).

This problem is multiplied many times over for mobile site development. Device independent rendering software solves a lot of the issues of screen size, memory limitations, paging etc – but what appears to be correct markup renders very differently on different devices. In my previous job, at a company that made device independent software, we had to make a large number of specific modules for certain devices to get round some of the stranger problems, one example that stands out is a device that would only display a whitespace after an anchor if you put three nbsp’s followed by a space.

Every device (even from the same manufacturer) seems to have different quirks displaying the markup but most of the problems occur in the different CSS implementations. Manufacturers either intentionally leave bits of the spec out or implement it incorrectly.

So, once you have your markup correct you have to start testing on a large number of devices to ensure your new big idea is displayed correctly. What’s the best way forward? There are several options:

  • Buy every device and test each one
  • Use emulators
  • Use a service such as DeviceAnywhere
  • Find a cheap way to hire lots of people to test on their own phones
  • Automate the testing process
  • Ignore it and hope everything is OK

Clearly buying every device is not an option given the number that exist and the rate of change. Normally devices are broken down into 5 or so different categories based on capability which gives a smaller sample size, but this can still be a large number of devices.

Emulators have their uses, but are not adequate for testing. Most of the emulators I have used are not accurate enough to test on and I have seen some that let markup through that crashes the real device.

Services such as DeviceAnywhere provide the actual devices to test on through a web interface, this is the best way of getting to a large number of devices but it is still time consuming to test on them all, and it can get expensive. There is also limited coverage at the moment so you can only test for a limited number of geographies. For a good overview of all the services available hop over to an article by an ex-colleague of mine, Tarek

If you know a lot of students you might be able to get some good testing coverage, but this would be tough to organise. A new service could address this problem, Mob4hire is like a social testing network which could be useful. There are also a number of options for this in the Far East, but network coverage could be an issue.

The holy grail here would be automating the testing process, but this is certainly a far off, if not impossible dream. You can automate testing the markup and this is useful for regression testing but you cannot automate looking at the display on the phone to check it.

If you think ignoring the problem and hoping it will be OK is the way forward, then you are probably in the wrong industry.

I would like to have been more positive in my first blog for bemoko and to reward you for getting this far with the answer to testing on mobile devices. Unfortunately I can’t provide that answer but bemoko has a lot of experience in testing sites that have been produced and, while it’s not perfect, a good strategy can be built to give a decent testing coverage using a mixture of the methods I have outlined above.

Share and Enjoy:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google Bookmarks

Posted in: mobile

Comments

  • Olaf Dunn

    Jan 30th, 2008

    I recieved a press release on the new mobile complete service that may be of interest to you, and to this article, so I thought I would share…
    http://mobilecomplete.com/news/jan29_dapro_eng.htm
    “Mobile Complete Introduces Mobile Test Automation through its DeviceAnywhere Service
    DeviceAnywhere delivers industrys first mobile test automation service across over 1000 real mobile devices worldwide”

  • Software Testing

    Apr 4th, 2008

    I found your blog via Google while searching for software testing and your post regarding mobile sites | bemoko’s mobile web looks very interesting to me.I just just wanted to drop you a note telling you how impressed I was with the information you have posted here. I also have websites & blogs so I know what I am talking about when I say your site is top-notch! Keep up the great work, you are providing a great resource on the Internet here!

  • kannan

    Apr 19th, 2008

    hai

  • Kevin

    Jul 31st, 2008

    I agree, it’s just not feasible to test against every possible device. My own feeling is that, in scenarios like these, it requires a combination of techniques to get good results.

    Regression testing of markup is good to test application logic and device-specific output.

    Designing sites to degrade gracefully means that the majority get a usable experience, albeit not always optimal. I believe that much of the value in mobile services is in utility rather than presentation.

    Testing key application requirements from within the application – for example, a setup process that asks questions about the user experience, such as “Does the title appear twice at the top of the screen?”, allows the capabilities database to improve continuously and automatically.

    Providing the means for users to report anomalies very simply from their phone allows the capabilities database to be improved manually.

  • Murali

    Jan 28th, 2009

    I’m new to bloggong & My blog is about basics of Software Testing,Manual Testing,SDLC,Testing Techniques,Levels of Testing,Types of Testing,Test Planning,Test Execution,Test Development,Bug Tracking,Result Analysis,Test Design Techniques and QTP. so I write about that which I know.Give it a visit if you get a chance..
    feel to free to visit:http://softwaretesting-guide.blogspot.com/
    Regards,
    Murali

Leave a Comment

Name (required)

Email (will not be published) (required)

Comment