UI and mobile test automation looks like low hanging fruit. Both may largely free up your teams from having to test manually. If mobile test automation is not used correctly the result will be a low return on your investment.
For example, if your tests are slow to develop, don’t function properly, are hard to maintain and take time to analyze then you will end up with poor results. Capgemini reported in their World quality report 2018-2019 that 52% of enterprises say that the biggest challenge in web and mobile app testing is not having enough time to test. Furthermore, 61% said that it is difficult to carry out mobile test automation because their applications change too much with each release.
In this article, I would like to return to the basics of how to make your automation project successful. I understand that the guidelines provided here in this post might not be applicable for all projects and that in some cases it may take up to a month to implement. Given the explosion in the automated testing tool market, now is as good a time as any to return to the basics of creating and running a successful mobile test automation project.
When we look at our customer base we can clearly see that it is a scale issue. The more you need to test and the faster you need to get to release the more tests you need. As you scale up in terms of the number of tests, and the types of functionality they cover it can lead to issues.
This is exactly what we are trying to eliminate by taking the steps below.
Object identification – don’t work around go straight
This is a core piece of UI test automation. In order to simulate user actions on an application you must first be able to identify each on-screen element. Simply put, you cannot automate tests for elements that the tests cannot identify.
The issue around identifying the elements in your application is not a technology issue, rather it is a communication problem with your R&D teams.
Let me explain.
Let’s say you have a new app that you need to test. (In this scenario you are both the application developer and the automation engineer.) Whenever you would like to identify an element you can add some unique properties to the element (ID, accessibility identifier …). Now the element identification issue disappears.
In reality, most automation engineers will just try to work around the identification issue by creating some heuristic. The problem with heuristics is that they tend to fail because they are usually overly complex. More importantly, they tend to relate to other parameters like page trees which tend to change across versions.
A few optional solutions:
- Automation engineers and R&D teams should create a well-run practice in which elements that need identification are defined along with how to identify them. When a UI is created one of R&D’s tasks is to create a unique identification for all of the required elements.
- Requests from automation engineers to add element identifications should always be the first R&D priority (on top of everything else). That way whenever you need a new identification, it will be available very quickly.
- The automation team works with the application code and adds each element’s identification themselves.
The third option above is my favorite and we have been following this practice internally very successfully.
Breaking Test Chains – any test any time
In your mobile test automation framework, you should be able to choose a test and execute it. You don’t need to worry about required preconditions, these are something that the test should handle. Tests should not be dependent on other tests or anything outside the test itself. We call these stateless tests and they can be executed at any time. Stateless tests function much better than a test chain in which tests are dependant on each other and need to be executed in a specific order.
Test engineers create test chains to avoid repetition and save time. They also come about as a result of a lack of test data management, and by the simple fact
that sometimes enterprises do not pay attention to test suite design.
The problem with test chains is as follows. Since each test relies on the tests that come before it if one test fails they will all fail. The result is that tests will become even harder to debug because you must debug all the tests in the chain. This leads to another issue that makes test chains a less than optimal testing method. Due to the massive amount of information you are testing and the order of the test chain, you must now contend with “hidden configurations”. A result of “hidden configurations” is that few people in your company will actually know and understand the full test configuration because there are so many tests and each test can change one small item.
We do have a few approaches to achieve stateless tests:
- Test data management – in most of the environment, using the right test data as a specific user would achieves the exact test state.
- Use an API call to build an environment from scratch (will work only if the user environment setup time is very fast).
- Adaptive tests. Build tests that will adapt themselves to the environment they are executing on.
The approach you should select is highly dependent on the system you are testing.
Small tests – API for setup and teardown
Ideally, every test should test the smallest possible application functionality. This requires a large number of tests to be created to generate the needed coverage. This is important because once your mobile test automation project scales, you will find yourself wasting a lot of time on execution analysis.
Small tests will ease the analysis process and that will take more and more of your time.
The problem is that most of the systems you are testing are stateful and in order to test something you need to perform some setup. The truth is that in most of the systems the setup required is much larger and more time-consuming than the test itself (that should be very small and fast).
A few optional solutions:
- Create tests to test the smallest possible functionality.
- Use an API to do anything that is not the core test functionality.
- Share setup between tests. The test will identify the state and if needed will perform the setup.
Business-driven – not UI driven
Build tests driven by the application business objective (and not the UI).
Take the example above.
A test on an app used for banking, for example, should look like this. Log in, get the user balance, make a payment, validate balance. In other words, as a user would interact with the application.
A UI driven test script, however, is given in a testing language that will not be immediately accessible to everyone who will need to see your test results.
Business-driven tests create functions that will “hide” the UI from the test itself. These functions can be then used in different places in your tests and can have a different implementation to different platforms, like mobile/desktop web.
The benefits of business-driven tests are:
- Easily readable by everyone (coders and non-coders alike) because it is written in a simplified language.
- Entire DevOps team can write tests – including business analysts, again due to the simplified coding language.
- Abstraction enables cross-platform reuse for consistency – Android/iOS/Web – there will be some changes based on implementation and platform you are using but the tests will remain the same.
- Minimal code repetition which will save time and reduce errors.
In far too many cases you cannot execute your tests due to some issues with your staging/QA environment. It also impacts every test failure analysis as sometimes identifying that the issue is an environmental one is not so simple.
In my work, I have experienced organizations in which the QA environment is available less than 20% of the time.
A couple of optional solutions:
- Execute your UI tests only after all of the API tests pass successfully. If you have a problem in your environment it is easier to identify as part of an API test, because you will know if the environment caused the error or not.
- (Recommended) Run your tests against a mock-up environment.
Continuous Execution – regardless of the application build cycle
There are different reasons to execute your mobile test automation tests. The main one is obviously to validate new changes in the application under test. Another very important reason to execute your tests is to verify that they are stable. Many test projects tend to neglect this part. When I talk about continuous test execution, I mean that once a test is created, we will start to execute it periodically regardless if we have a new application or version to test or not.
When we talk about continuous execution the execution will run whether you have a new commit or not. This helps to stabilize as it is a time-consuming process. In our experience, it could take around 1000 times for a test to be run before we would consider it “stable”. It is only after 1000 successful executions of a test that you know it is stable and if it fails it is due to an issue in the application under test. Now if you execute your tests once a day, repeating it 1000 times will take around 3 years.
The mobile test automation process is in and of itself code that needs to be tested. Running tests continuously makes sure that our testing environment is running smoothly even if there is no new version being tested. This means that if there is an error when testing a new app version we will know that the error is from a problem with the application itself and not the testing environment.
Some suggested solutions are:
- Once a test is created run it on the existing version to test it.
- Run whole test projects periodically, even when there is no new code, to make sure the test suite is functioning properly.
- Fix test suites quickly in order to be ready for any new application versions that will be coming out.
Analyze quickly – be ready to scale
Mobile test automation is evolving. When we started the challenge was: How do I click on this element? Then the challenge evolved to: I have created many tests how do I execute them in parallel on tens or hundreds of devices/browsers?
In the last 2 years, it changed again: This is an example from a real customer of ours. “I am executing my tests every hour, In 24 hours I execute around 1 million test cases. Every 1-hour execution can take me a few hours to analyze. How can I understand the root cause quickly?”
We have found that with actionable analytics you will be able to drive decisions. Our test analytics capabilities provide the following that makes test analysis a lot easier and more effective.
- Clear sharable reports
- Consolidated test results for analysis
- Dashboards for management decisions
Wrapping up the mobile test automation basics
Digital transformation has changed a lot of the way we work. It has especially changed the way that we perform UI testing. It is not that the basics have changed on a fundamental level, but they have shifted to the left, which makes looking at the process and methods by which we perform web and mobile application testing important. These 7 steps will help you create a more effective and efficient UI testing system. We have learned these valuable lessons, not simply from talking to our customer. We at Experitest also use a lot of these methods and can attest to their effectiveness.
And If you would like to try any of the mobile test automation methods above the SeeTest Continuous Testing platform will give you easy test creation tools that will help you create stable tests and carry out large-scale parallel execution. It comes bundled with an analytics tool for web and mobile as well. Sign up for a free trial, now.