The pace of technology is moving so fast businesses have gone from incremental transformation to continuous transformation.
The life cycle to plan, develop, test, and deploy any application has been reduced from months to weeks. Mobile testing cannot be “we take your build, test it, and get back to you in a month.” Mobile testing, just like every other step in the development and operations (DevOps) process, must be continuous for a business to be competitive.
Mobile testing teams, along with the mobile testing tools they rely on, must adapt to a continuous process.
As the proportion of the IT budget dedicated to quality assurance continues to rise, so does the need to reorganize the QA process itself to make sure these resources are being used productively. DevOps itself is the reaction to the need for a continuous development and testing process. From this process comes the need for Agile. Most businesses still don’t get it.
Agile comes from the new demand that mobile applications be updated daily, developed weekly, and incremental development and testing is no longer possible to handle such speeds. One group can’t “do its thing” for a month, then hand it off to the next group to “do its thing” for another month. DevOps is the Agile process of everybody working in tandem to get the most updated version of your company’s mobile app to market as soon as possible.
The rate of innovation for all of the platforms apps are functioning on is making mobile testing very challenging. The standard mobile test coverage is well over 50 million possible combinations of mobile devices, operating systems, and local networks. Once you include variables like the latency and throughput of a local connection, or the versions of each mobile operating system the test coverage becomes exponentially massive.
But it must be covered. Test coverage is all about pleasing the customer by optimizing the user experience. The more the world is interconnected, the more dangerous a single bad experience can be. Conversely, the happier the customer, the better the brand ambassador he or she becomes. User experience is the end game of mobile testing. This requires both function, to make sure everything is working properly, and performance, that everything is working smoothly.
Agile Development Tools Necessary for Mobile DevOps
To achieve Agile, you need to integrate all elements of the process. This requires continuous integration, and a CI platform that does exactly that. The core of continuous integration is automation, and end to end testing.
In order to optimize your testing ops, combining it with the rest of continuous integration, you need test automation, cloud testing, and complete integration with all the major continuous integration platforms.
The first piece of the puzzle is the cloud. A solution like SeeTestCloud gives you the ability to create your own private mobile laboratory, and to give access to all of the devices in that laboratory to all the developers and testers in your DevOps team.
For a system integrator, you can establish a mobile lab for your clients. If you are a business, you can set one up for your own internal use. The right mobile cloud testing tool operates safely within your company’s virtual private network (VPN), meeting all security requirements by your IT security team. All developers and testers gain access to the devices they need. In combining the tools that both developers and testers use, you realize immediate productivity gains.
The cloud is the heart of optimizing a continuous testing process by allowing anyone to access as many devices as you need at any point in the day. You are no longer limited to devices plugged into your local USB port or the local server. A QA manager can give access and permissions to each tester as to which devices are available to them and when. He can set priority to testers working on different projects.
SeeTestCloud is a mobile cloud testing tool that has an application repository where testers can decide which version or build of the application they are going to test on. The QA manager also has the privilege to gain or restrict access of each application or version to each tester for a given project to ensure the right builds are being tested.
An administrator can set up an automatic device clean up procedure. This is where there are large groups of users using your apps and you want to make sure that each device will clean out the app from the device when the next tester or user accesses it to maintain security of unreleased mobile applications.
One of the advantages of unifying the tools and assets developers and testers use is the speed at which developers can reproduce failed tests. Quite often a tester will report a defect, and the developer will be unable to reproduce the exact scenario that led to the defect. This happens primarily because developers and testers use different devices to work on. By using the same pool of devices, the scenarios will be available to the developer once the tester identifies the defect, saving the developer, and the DevOps team a lot of time.
For both SeeTestManual and Experitest’s SeeTestAutomation, a report is generated detailing every action performed in the test, with screen shots, and whether or not the test passed. Links on the report enable a developer to click the link, and go directly to the device, and the scenario of the specific step.
How Automated Mobile Testing is the Backbone of an Agile Development Lifecycle
Without automation there is no Agile.
Today’s mobile application has so many functions and capabilities, you cannot test accurately for all of it without automation, and parallel testing. Experitest’s SeeTestAutomation has both. It enables you to create an automated process, testing to a greater scale, and integrating into a continuous integration platform to automate as much of the DevOps process from development to deployment as you can.
It all starts with creating a single test script. You can manually code a script, or you can call up any device from your computer or the cloud. Once you do, a reflection of the device’s screen will appear on your desktop and you can use the device with your keyboard and mouse. The sequence of actions you perform will make up the test script as Experitest’s SeeTestAutomation records each action you make. These scripts can automatically translate into any other major programming language used in mobile testing. SeeTest supports a wide variety of programming languages, testing frameworks, and continuous integration builds. You can use or convert your scripts into Java, HP Tools, Visual Studio, Python, Ruby, and more.
Once you have the script mobile test automation is a snap. Here is the process:
Step 1: After a bug is reported from the tester to the developer or once the developer completes a new version of the mobile application, the developer puts the new code into a source code repository.
Step 2: Using the new code, the Continuous Integration server builds the application.
Step 3: The test scripts that the mobile testing team prepared are automatically executed against the mobile application. It is done in parallel execution on multiple devices to maximize test coverage and minimize total lifecycle duration.
Step 4: The test results are generated and sent to testers and developers.
Step 5: If the tests pass, the mobile application is deployed to market.
Step 6: If the tests fail, the results are sent back directly to the developer. He or she will update the code, and reload it into the source code repository where it will go back to the CI server to build the app again and restart this process.
The key to it is on-the-fly instrumentation. Any process that demands a developer create an instrumented build is not automatic. An automation tool must have on-the-fly instrumentation to automate every step in the testing process. If a tool claims to be part of a CI process, you must check if it has on-the-fly instrumentation.
Parallel Test Execution and Dynamic Allocation of Mobile Devices: Major Time Savers
Sequential execution, or running your test scripts on one device at a time is no longer feasible. Parallel execution helps you ensure that your mobile applications are running smoothly on the maximum amount of devices and operating systems in the minimum amount of time.
Experitest’s SeeTestAutomation and SeeTestManual both enable parallel testing. Used in conjunction with SeeTestCloud, you can execute your test scripts on over 100 devices all at once while you are eating dinner.
You can set up your CI platform to run the test scripts over the devices you select in your mobile device lab. Once the test results come back, the CI platform will move to the next step in the process.
But what happens if the exact devices you need from the mobile lab aren’t available? Will your process stop automatically until they are ready? That can cause a serious bottleneck in your mobile testing! This is where a dynamic allocation of devices feature is important.
Available on SeeTestCloud, you can broaden the way you request devices by setting a parameter, rather than asking for a specific device. You can request a Samsung Tablet, and the testing tool will see which are currently available. You can request a Smartphone running on Android Lollipop 5.1 or later. The devices that match these parameters will be the ones the CI server executes your tests on. This reduces waiting time for devices, shortening your development to deployment life cycle. It optimizes the use of your mobile lab, ensuring a maximum return on the investment your IT department makes in its mobile application testing.
What to Do? Adding Performance Testing to the Mix is Vital but it Expands Test Coverage
Performance is the ghost of functional testing. It can remain invisible throughout the entire mobile testing process only to haunt you when you least expect it.
This is due to most mobile testing being carried out in state of the art IT departments. A mobile tester will conduct his functional tests in an ideal environment with a first rate local connection. A typical user, who doesn’t have access to the same resources as the multi-million dollar IT department, will use your application with a normal connection where performance issues are more likely to occur. If a page takes too long to appear, or a function takes too many seconds to execute, the user will assume that the application is broken and delete it.
The right set of performance tools can prevent this from happening.
SeeTest Network Virtualizaiton enables you to create different network conditions to see how users of your mobile applications on various mobile devices react to varied local networks. You can recreate your own network profile which mimics what they are using, rather than the ideal environment you test in, but that doesn’t catch these issues.
A QA manager can generate a set of network profiles, mimicking the most likely locations your users will access the app. The profiles can be made by a wizard, or they can be customized from scratch to simulate a user’s location, connection, technology, and device signal quality. Profiles can also be customized to define throughput parameters (bandwidth and latency) as well as packet loss levels.
Put it All Together and the Results Are Real
When you combine test automation with performance using a mobile lab based on cloud testing, while integrating it all with a continuous integration platform you are now in the ideal position to maximize your test coverage while minimizing your development to deployment lifecycle.
Using SeeTestManual or Automation, you can go back to your test’s and change the network profile. You can go back to specific test cases (scenarios), and test again, but under different local connections to see how the functions perform. Are they performing fast enough under every network you expect your users to access your mobile application over?
A recent case study revealed that using automated mobile testing tools in conjunction with a continuous integration server enabled a large financial institution to reduce their development cycle time by 30%. They were able to reduce parts of their regression testing cycles by 70%. Overall testing costs were reduced by 40%.
Get Updates on Software Quality Assurance at
our Experitest LinkedIN Page