Test Faster and Smarter With a Mobile Lab

Share this knowledge!

Any enterprise developing mobile applications can significantly benefit from establishing an internal mobile device lab, increasing both productivity and convenience of the development and quality assurance process.

This report provides a guide on how to build such a lab, detailing the advantages, basic requirements, and demands that should be taken into account.

The Next Step in Mobile Testing Efficiency

Companywide mobile projects keep expanding, increasing the number of bodies involved in development and testing. As more devices come to market, the quality of these applications must be verified over an ever-growing variety of screen sizes and resolutions. To ensure that an application performs equally well on all device sizes, both in terms of functionality and user interface, a large number of devices are required to test the appearance and functionality of the application.

A typical project would require double the number of devices for the number of people working on it. For example, for a project with 20 people, one will need around 40 devices. For more than one project, the number of devices needed can grow exponentially. This can become quite costly but on the other hand, buying fewer devices creates availability issues, arguments and project delays.

Mobile device labs are a great solution for such problems, as they keep costs low while still making devices available for teams throughout the company.

MOBILE DEVICE LABS MEET ALL OF THE FOLLOWING NEEDS AND MORE

  1. They make devices available for anyone at any time
  2. They centralize all devices into one or a few locations
  3. They enable easy identification of a particular device with a specific operating system, version, and manufacturer
  4. They provide world-wide access to teams working from multiple global locations
  5. They allow for proprietary work to be done by one team, while being kept from other teams, working on different projects
  6. They keep the devices in a secure location, w where they are not physically handled

What Your Mobile Lab Can Accomplish

ANSWER THE NEEDS OF ALL FOCAL POINTS

Using Cloud-based software you can quickly establish a mobile lab and answer the varying needs of all focal points involved with a project.

PROJECT MANAGER

Needs to be able to determine who uses what devices and when. The project manager needs to prioritize projects, reassign devices and also reserve devices to testers located in different time zones, to ensure a 24 hour continuous testing environment for the applications being primed to market.

APPLICATION DEVELOPER

The application developer needs to access devices and control them. A mobile lab enables him to see a reflection of the device on his screen while he manipulates the device by his local keyboard. He can remotely deploy his application to create new builds and debug the applications as he sees fit.

QA MANAGER

Needs to reserve devices for his team, as well as for his own Quality Assurance. Needs to work with the application developer to anticipate when new builds or versions will need to be tested. The QA manager also reports what worked well and what didn’t on each specific device, to enable the developers to fine-tune the applications to work as well on some devices as it did on others.

DEVICE LAB MANAGER

The only member of the company that should have permission to physically touch the devices. He can manage the availability of the devices for all project managers. If necessary, he has the authority to override a device reservation in favor of a project or task of critical priority. The manager can control the specific versions of operating systems for each of the devices, enabling applications to be tested on older versions. He can produce usage reports in order to know which new devices to buy, maintaining an adequate subset of devices relative to the universe of total test coverage.

MANUAL QA USERS

Need to be able to access a specific device, or a number of devices in advance. Need immediate knowledge of which devices are available, which are taken, and when they can reserve the devices they need. We need to automatically report bugs into a QA tool, including device logs and screenshots.

What Your Mobile Lab Can Accomplish

AUTOMATION DEVELOPERS

Need constant access to which devices are available, what operating system they use, the browser. They need to integrate scripts with the devices as well as the timetables available to execute scripts needed to speed up the testing process, especially for regression testing.

SECURITY OFFICER

Needs to verify applications are not exiting the organization’s firewall. All operations by development and testing teams must be performed securely inside the company’s VPN. The security officer needs to verify that all test data like account names, users, passwords, stay securely within company databases.

Finding the Right Solutions

KISS – KEEP IT A SIMPLE SOLUTION

Solutions should return your investment easily and quickly. A software solution achieves exactly that. Look for something you can download immediately and use within minutes. Make sure it is something that does not require you to use devices outside your company’s network and might compromise your security.

THE LAB SHOULD HAVE MINIMAL IT REQUIREMENTS

Getting new software approved by any enterprise IT department takes time. A solution with minimum requirements can speed up the process. Requirements for a mobile Cloud can be as simple as this:

  1. It can be installed on a single machine.
  2. You just need to have a single port open on your server.
  3. Need to install needed USB drivers.

Here are the IT requirements you should avoid:

  1. You have to use an outside location for your lab. This defeats the purpose of setting one up! The devices you access are outside the company firewall. You lose control over your own mobile applications. You expose your company and your users to surprises the vendor can unexpectedly throw at you. You also can’t decide which types of devices you get to test on.
  2. The vendor needs to ship your hardware that requires installation before using it. This is guaranteed to dramatically slow down the approval process from your IT department. It compromises company security by installing outside hardware into your organization’s network.
  3. Administration rights need to be set to the minimum. This is also a serious security issue.

THE IMPORTANCE OF A SOFTWARE SOLUTION

When you need to scale you cannot sit and wait for special hardware to arrive or installation to be complete. Choosing a solution that isn’t downloadable on-demand takes away your testing flexibly.

Can you pay to scale up? How much will it cost, in time and resources, to procure the devices you need for all the projects currently running? How much more will it cost as your business expands, running more projects in your mobile application’s operations? How flexible is your mobile lab to take on more devices to accommodate a rapidly growing mobile market and test coverage?

Background

A large international bank with R&D centers in Europe, India, and the US developed 15 applications for its retail users. Another 5 applications were developed for bank representatives for internal use. These applications were all developed for iOS, Android, Blackberry, and WindowsPhone handsets.

The banks’ mobility group-size was around 500 people.

Originally their development resources consisted of ‘Boxes’. For every application they were developing, there was a dedicated ‘Box’ containing around 70 devices. When a developer or tester needed a device, they would need to locate the ‘Box’s’ manager, take out the device, and return it to the ‘Box’ when they were done.

Challenges Created by the Old Solution:

  1. For 20 applications, each with a ‘Box’ of 70 devices, the company needed to buy 1,400 devices.
  2. It was impossible to quickly identify a specific device with a specific OS and version. A developer or tester had to turn on every device to see what it was running on until he found the right one.
  3. Devices weren’t charged regularly so after successfully locating a device additional time would be wasted to charge it.
  4. Devices had a habit of disappearing. Company data remained on those unaccounted devices. Security teams could not access the devices, or erase whatever data remained on them.
  5. People forgot to wipe the application and its data. This information was accessible by anyone using the device. Developers working on different applications had open access to everything.
  6. It was hard to track what bug occurred on what device. If a small error was made, the developer or tester had to remember what device it was made on. They could spend hours sifting through ‘Boxes’ and scores of devices inside them to remember which one had the minor issue.
  7. Constant physical handling meant lots of wear and tear on the devices themselves.

The Solution: A Mobile Lab

As costs went over budget and deadlines were constantly being missed, the bank decided to build a mobile lab. Three dedicated locations were selected: one in Europe, one in India and one in the US.

A set of 200 devices was selected. The device matrix was build based on market demands – 120 devices represented the most used devices, 80 devices targeted noncommon devices. Many of the devices were among the most popular in emerging markets.

Since the organization’s existing infrastructure for developing mobile applications worked well setting up a mobile lab did not create any need to change it. In each of the three locations, only one person had physical access to the devices.

As the bank’s business expanded, so did its mobile applications. As more devices became available on the general market, their device matrix expanded accordingly. Their mobile lab grew to more than 300 devices on the Cloud, all available to every developer, tester, and quality assurance engineer.

Everyone was happy:

  1. Developers no longer needed to chase after a specific device when they wanted to reproduce bugs. They could reserve a specific timeslot in advance and access “their” device during that time.
  2. QA managers could plan which devices they were going to use in advance. They could control the tested application and they see the progress.
  3. The device lab manager administrated all the devices, users, and reservation times in less than one hour each day. He assigned the same project to team members at offices in US, Europe, and India creating a continuous testing environment. The mobile applications were tested 24 hours a day.
  4. Manual testers used their keyboard instead of having to manually press in a set of commands. A reflection of the mobile device, or devices they reserved appeared on their computer screens and ran whatever commands they wanted remotely.
  5. The automation engineers ran automated tests from their desktops. They could pre-determine when the devices would be available, which devices they wanted, and how each script would run for each device. Reports informed them of what was working and what wasn’t. For a bank, with a large amount of legacy software, and the demand that their applications run perfectly there is a great need for regression testing. Automation engineers could run regression tests on certain devices, reserving them at the same time each day to make sure all new enhancements to the banking applications ran perfectly for every level of code, and every device that code was executed on.
  6. The security team ensured that every project was safely conducted within the company’s firewall. Nothing went beyond the network without their knowledge, approval, and supervision. Applications tested on specific devices stayed with the application teams, they weren’t available for groups working on other applications. The highest level of security was maintained at every step.
  7. Purchasing over a thousand fewer devices saved real money. Reduced wear and tear of each device meant less replacement. A continuous testing environment brought each application faster to the market, empowering customers to always have the most cutting edge version available. Customers were excited to do more business with the bank, as people were attracted to the apps themselves, and new business poured in. More business meant more revenue.

Overall, there were fewer problems with the application. Customers were satisfied and continue doing business with the bank, knowing they can trust the bank to have the best mobile applications on the market.

Conclusion

The rapidly expanding number of devices on the market demand more testing. More functionality is available for every mobile application also demands more testing. A quicker development to deployment cycle means this added testing must be done in less time. A mobile lab is the best solution. It saves money. It enables more thorough testing. It does all this while keeping your testing ops efficient.

To download the full whitepaper click here.

Guy Arieli, CTO, Experitest

Share this knowledge!