September 2, 2020 Jonny Steiner

These are the 7 best Mobile Test Automation Frameworks in 2020

Share this knowledge!

Well, 2020 has not really panned out the way most people have expected. At least through the first half of the year. Global pandemics forcing people to work from home, civil unrest worldwide. It is both scary and incredible how this is all going on simultaneously. That is not to say that it is all bad news. Let’s take the world of mobile application testing and the mobile test automation frameworks used in it. Not only has the exponential growth we have seen in the past few years not slowed it seems to be ratcheting up even more.

Once the lockdowns and closures began due to the overwhelming spread of Covid-19 many of our customers sent their software testers home. I am so proud of our company that we were able to not only support them working remotely but in many cases improve their automated testing even from afar.

Mobile Applications are Thriving

Let’s start with rattling off a few numbers courtesy of Statista. There are an estimated 3.5 billion smartphone users worldwide. These users, spend 90% of their time on their phones using apps. Just how many apps vary by the operating system but there are 2.2 million apps available on the Apple App store, and 2.8 million on the Google Play Store. Truly huge numbers that become even more eye-opening when you figure in the fact that every single one of those apps needs to be tested.

Needs of Testing

When you release an app in 2020 it has to work, that much is obvious. Testing your app before release is a given since you want to provide your users with a UX that keeps them engaged an using the app. However, in 2020 it’s not enough to make sure your app works. It has to work on any number of devices running a vast range of OSes. But wait! There’s more. An app that works must also not consume too much of a device’s memory or battery life. And it all needs to come together because, if your app does not perform to the standards of the public, that is to say flawlessly, then it will sink like a stone.

Automation guiding the way

We have seen through ½ of 2020 that the mobile app testing market is shifting dramatically towards automation. Quality Assurance being as important as it is makes automation tools and frameworks central to the process of testing and releasing apps. This is only going to help the app testing industry moving forward.

Testing on Real Cloud-Based devices

The statistics say that this year there will be as many as 1.5 billion smartphones active worldwide. Within these devices are different OSes, hardware, screen resolutions, and a lot more. That makes testing an app on all devices almost impossible. Unless you have something like a real-time accessible cloud with a number of devices ready for testing via cloud technology.

We have good news on that front because that is EXACTLY what we have. Using the SeeTest Digital Assurance Lab by Digital.ai you will be able to log in from your browser wherever you are and access any number of remote devices located in one or more of our 8 global datacenters. The best part is the ability to obtain fast feedback and test devices beyond simple functional testing. We also provide the tools to test device hardware and network conditions.

The Growth of DevOps and Agile

Organizations need to make rapid changes to their apps. We are seeing a shift where Agile development is being combined with DevOps in order to set the processes for mobile apps. This combo is almost unbeatable in the way that it is helping mobile app development delivery move faster and with higher quality.

With our understanding of the current state of mobile app testing let’s take a look at the 7 best mobile test automation frameworks to get you the results you need.

 

mobile test automation frameworks

Galen

The Galen Framework is open-source software that gives testers the ability to execute automated tests for building responsive web apps. The tests are written in Java (which is compatible with SeeTest). The Galen Specs language describes how elements align and their position on a given page. Based on Selenium the tests are executed via the Selenium Grid if you want to execute cross-browser testing.

Galen is also fully integrable into our SeeTest Digital Assurance Lab so you can test on 1000+ different browsers and real cloud devices running a number of different operating systems. You can read more about Galen here.

Galen Pros:

  • No other framework provides as many features specifically for UI Automation.
  • Pixel to pixel image comparison
  • Multiple browser support
  • Grid Integration
  • Developer support

Galen Cons

  • Element identification must be defined in both a spec and a pages file.
  • Hard to have to refer to elements in more than one place when functional testing.

 

mobile test automation frameworks

WebdriverIO

Growing in popularity recently WebdriverIO is a web and mobile automation framework. It is open-source and written in JavaScript running on Node.js. Testing with WebdriverIO enables you to test web apps and mobile apps.

The reason for WebdriverIO’s popularity is a result if its Selenium backing. That means that you can run tests on all browsers. These advantages are furthered by the elimination of writing Java-based scripts. You can also integrate it with many different testing frameworks including our own SeeTest. You can read more about the solution here. 

WebdriverIO Pros:

  • Simple Syntax
  • Easy to Start
  • Fast Configuration and setup
  • Community Support

WebdriverIO Cons:

  • Api Documentation is incomplete
  • Hard to make Assertions at first
  • Debugging is only done through WDIO task runner

 

mobile test automation frameworks

Cypress

For end-to-end test automation, Cypress is best suited to modern web app environments. Built as a JavaScript framework it also runs on Node.JS. With Cypress, writing integration, unit, and end-to-end tests is easier than ever.

The packaged Test Runner is an open-source and a cinch to set up and use. Once your tests are ready you can integrate with the CI tool of your choice and record with the Dashboard. The best part is that you don’t have to install any server dependencies or drivers. The framework is extremely developer-friendly and the tests are easy to understand. We have a full review here.

Cypress Pros:

  • Good documentation
  • Large community
  • Fast and reliable
  • Parallel execution
  • Cross browser support

Cypress Cons:

  • Need to be a pro-level coder
  • Tests often lack reliability
  • Large package size

 

mobile test automation frameworks

TestProjectIO

This cloud-based test automation framework allows users to create automated tests for their web and mobile applications. Create tests for Android or iOS operating systems, and all web browsers are supported. Tests can be managed and shared from the TestProject IOs web app to team members. REST API give the capability to integrate with your CI workflow. Another great aspect of the tool is that it is community-powered and used to develop, record and breakdown your test automation.

TestProjectIO Pros

  • Collaborative environment
  • User Friendly
  • Community Support
  • Compatible with different platforms

TestProjectIO Cons

  • Only supports the Java programming language.
  • Stopped agent does not show any warning or notification.

 

mobile test automation frameworks

Cucumber

Based on the principals of Behavior Driven Development (BDD) the Cucumber framework is used to write acceptance tests for web apps. Automate functional tests using easily readable and understandable formats. Cucumber has been expanded from its origins using Ruby to also include Java as a language. The goal is to use Cucumber in order to test the system rather than the individual lines of code. We also have a full overview of this tool. 

Cucumber Pros

  • Gherkin language is easy to use and understand.
  • Map steps in any programming language.
  • Define method test case priorities
  • Easy to maintain and reuse tests.

Cucumber Cons

  • Gaps when unit testing with cucumber.js scripts
  • Requires external reporting integrations
  • Documentation lacks detail

 

mobile test automation frameworks

Serenity

With a similar BDD philosophy to Cucumber, Serenity acts as a wrapper for Selenium that also supports working with APIs. It is open-source software that aids in writing better-automated acceptance tests. Serenity can also create live documentation for tested features that report on test results as well as the specific features tested. We have reviewed Serenity in depth here.

Serenity Pros:

  • Quick and easy documentation creation post-testing
  • Collaboration between business and technical roles
  • BDD methods for better test specifications and acceptance
  • Target tests at specific functional areas

Serenity Cons:

  • Feature Files take time to create and maintain
  • BDD documents need maintenance
  • Takes time to write automation code

 

mobile test automation frameworks

Robot Framework

Robot is a great open-source option for acceptance testing and acceptance test-driven development. External libraries are supported, and you can use the open-source tools for automation. Selenium is the most popular library used in web development & UI Testing. Use the text editor of your choice and the Robot Integrated Development Environment (RIDE) tool for writing your tests. The framework is built using Python and runs using Jython (JVM) and IronPython (.NET). We have a full review of Robot here.

Robot Pros:

  • Easy to read test cases
  • Reusable keywords in different tests
  • Test case and keyword development can be distributed
  • Supports different programming languages

Robot Cons

  • Keyword development takes time at first
  • Learning a new programming language or framework takes time

 

Conclusion

We see the technology in web and mobile app testing improve on an almost week to week basis. It is all there to help organizations save money and improve their general workflow. Of course, there is hesitation in the face of new mobile test automation frameworks, but as they rise in popularity and testers become more used to them their adoption grows. We looked at the mobile test automation frameworks above and we expect them to become trends in the industry over time.

Jonny SteinerContent Manager

Share this knowledge!