Well, 2020 has not really panned out the way most people have expected. At least through the first half of the year. Global pandemics forcing people to work from home, civil unrest worldwide. It is both scary and incredible how this is all going on simultaneously. That is not to say that it is all bad news. Let’s take the world of mobile application testing and the mobile test automation frameworks used in it. Not only has the exponential growth we have seen in the past few years not slowed it seems to be ratcheting up even more.
Once the lockdowns and closures began due to the overwhelming spread of Covid-19 many of our customers sent their software testers home. I am so proud of our company that we were able to not only support them working remotely but in many cases improve their automated testing even from afar.
Mobile Applications are Thriving
Let’s start with rattling off a few numbers courtesy of Statista. There are an estimated 3.5 billion smartphone users worldwide. These users, spend 90% of their time on their phones using apps. Just how many apps vary by the operating system but there are 2.2 million apps available on the Apple App store, and 2.8 million on the Google Play Store. Truly huge numbers that become even more eye-opening when you figure in the fact that every single one of those apps needs to be tested.
Needs of Testing
When you release an app in 2020 it has to work, that much is obvious. Testing your app before release is a given since you want to provide your users with a UX that keeps them engaged an using the app. However, in 2020 it’s not enough to make sure your app works. It has to work on any number of devices running a vast range of OSes. But wait! There’s more. An app that works must also not consume too much of a device’s memory or battery life. And it all needs to come together because, if your app does not perform to the standards of the public, that is to say flawlessly, then it will sink like a stone.
Automation guiding the way
We have seen through ½ of 2020 that the mobile app testing market is shifting dramatically towards automation. Quality Assurance being as important as it is makes automation tools and frameworks central to the process of testing and releasing apps. This is only going to help the app testing industry moving forward.
Testing on Real Cloud-Based devices
The statistics say that this year there will be as many as 1.5 billion smartphones active worldwide. Within these devices are different OSes, hardware, screen resolutions, and a lot more. That makes testing an app on all devices almost impossible. Unless you have something like a real-time accessible cloud with a number of devices ready for testing via cloud technology.
We have good news on that front because that is EXACTLY what we have. Using the SeeTest Digital Assurance Lab by Digital.ai you will be able to log in from your browser wherever you are and access any number of remote devices located in one or more of our 8 global datacenters. The best part is the ability to obtain fast feedback and test devices beyond simple functional testing. We also provide the tools to test device hardware and network conditions.
The Growth of DevOps and Agile
Organizations need to make rapid changes to their apps. We are seeing a shift where Agile development is being combined with DevOps in order to set the processes for mobile apps. This combo is almost unbeatable in the way that it is helping mobile app development delivery move faster and with higher quality.
With our understanding of the current state of mobile app testing let’s take a look at the 7 best mobile test automation frameworks to get you the results you need.
The Galen Framework is open-source software that gives testers the ability to execute automated tests for building responsive web apps. The tests are written in Java (which is compatible with SeeTest). The Galen Specs language describes how elements align and their position on a given page. Based on Selenium the tests are executed via the Selenium Grid if you want to execute cross-browser testing.
Galen is also fully integrable into our SeeTest Digital Assurance Lab so you can test on 1000+ different browsers and real cloud devices running a number of different operating systems. You can read more about Galen here.
- No other framework provides as many features specifically for UI Automation.
- Pixel to pixel image comparison
- Multiple browser support
- Grid Integration
- Developer support
- Element identification must be defined in both a spec and a pages file.
- Hard to have to refer to elements in more than one place when functional testing.
The reason for WebdriverIO’s popularity is a result if its Selenium backing. That means that you can run tests on all browsers. These advantages are furthered by the elimination of writing Java-based scripts. You can also integrate it with many different testing frameworks including our own SeeTest. You can read more about the solution here.
- Simple Syntax
- Easy to Start
- Fast Configuration and setup
- Community Support
- Api Documentation is incomplete
- Hard to make Assertions at first
- Debugging is only done through WDIO task runner
The packaged Test Runner is an open-source and a cinch to set up and use. Once your tests are ready you can integrate with the CI tool of your choice and record with the Dashboard. The best part is that you don’t have to install any server dependencies or drivers. The framework is extremely developer-friendly and the tests are easy to understand. We have a full review here.
- Good documentation
- Large community
- Fast and reliable
- Parallel execution
- Cross browser support
- Need to be a pro-level coder
- Tests often lack reliability
- Large package size
This cloud-based test automation framework allows users to create automated tests for their web and mobile applications. Create tests for Android or iOS operating systems, and all web browsers are supported. Tests can be managed and shared from the TestProject IOs web app to team members. REST API give the capability to integrate with your CI workflow. Another great aspect of the tool is that it is community-powered and used to develop, record and breakdown your test automation.
- Collaborative environment
- User Friendly
- Community Support
- Compatible with different platforms
- Only supports the Java programming language.
- Stopped agent does not show any warning or notification.
Based on the principals of Behavior Driven Development (BDD) the Cucumber framework is used to write acceptance tests for web apps. Automate functional tests using easily readable and understandable formats. Cucumber has been expanded from its origins using Ruby to also include Java as a language. The goal is to use Cucumber in order to test the system rather than the individual lines of code. We also have a full overview of this tool.
- Gherkin language is easy to use and understand.
- Map steps in any programming language.
- Define method test case priorities
- Easy to maintain and reuse tests.
- Gaps when unit testing with cucumber.js scripts
- Requires external reporting integrations
- Documentation lacks detail
With a similar BDD philosophy to Cucumber, Serenity acts as a wrapper for Selenium that also supports working with APIs. It is open-source software that aids in writing better-automated acceptance tests. Serenity can also create live documentation for tested features that report on test results as well as the specific features tested. We have reviewed Serenity in depth here.
- Quick and easy documentation creation post-testing
- Collaboration between business and technical roles
- BDD methods for better test specifications and acceptance
- Target tests at specific functional areas
- Feature Files take time to create and maintain
- BDD documents need maintenance
- Takes time to write automation code
Robot is a great open-source option for acceptance testing and acceptance test-driven development. External libraries are supported, and you can use the open-source tools for automation. Selenium is the most popular library used in web development & UI Testing. Use the text editor of your choice and the Robot Integrated Development Environment (RIDE) tool for writing your tests. The framework is built using Python and runs using Jython (JVM) and IronPython (.NET). We have a full review of Robot here.
- Easy to read test cases
- Reusable keywords in different tests
- Test case and keyword development can be distributed
- Supports different programming languages
- Keyword development takes time at first
- Learning a new programming language or framework takes time
We see the technology in web and mobile app testing improve on an almost week to week basis. It is all there to help organizations save money and improve their general workflow. Of course, there is hesitation in the face of new mobile test automation frameworks, but as they rise in popularity and testers become more used to them their adoption grows. We looked at the mobile test automation frameworks above and we expect them to become trends in the industry over time.