Experitest Blog

How to Select the Right Test Automation Framework for you

What is a test automation framework?


The more software systems increase in complexity as new development practices emerge, the more the quality assurance and software testing requirements rise. Now, among a number of testing methods, automated testing is the most common candidate to enhance your quality assurance process. At first glance, automated testing may look like a very difficult and resource consuming choice. But, diving into the matter deeper, you may find out that there are a number of established practices, rules, tools and libraries that significantly simplify the process of designing your test cases – test automation frameworks. Basically, test automation frameworks are tools for empowering and simplifying automated testing.


Test automation frameworks provide a number of benefits: lower expenses, lower entry threshold for testers, more readable and reusable code.


However, there are a lot of test automation frameworks available. In this article I’ll cover the most popular ones, that have rightfully become test automation industry standards:



Each of these testing frameworks provides its own benefits and applies its own limitation to the test automation process. The differences include supported programming languages, test case organization, a level of applicability. I will describe the benefits and limitations of each framework in details below. The goal of the article is to make it clearer which tests automation framework will meet your needs.



junit - test automation framework

JUnit is a test automation framework for the Java programming language. As one of the oldest, yet constantly evolving frameworks it holds a leading position in the world of automated testing. As it utilizes Java, it has one of the biggest benefits – cross-platform support.


In JUnit, test cases are represented by Java objects, where a simple annotation system manages the execution of test methods. Like in the code below:


import java.net.MalformedURLException;
import java.net.URL;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.remote.MobileBrowserType;
import org.junit.*;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
public class AndroidWebTest {
private static final String ACCESS_KEY = System.getenv(“SEETEST_IO_ACCESS_KEY”);
private static final String CLOUD_URL = “https://cloud.seetest.io:443/wd/hub”;
private static final String TITLE = “Testing Website on Android Chrome with Java”;
private AndroidDriver driver = null;
public void setUp() throws MalformedURLException {
DesiredCapabilities dc = new DesiredCapabilities();
dc.setCapability(“testName”, TITLE);
dc.setCapability(“accessKey”, ACCESS_KEY);
driver = new AndroidDriver(new URL(CLOUD_URL), dc);
public void testAppiumOnChrome() {
if (driver.getCapabilities().getCapability(“device.category”).equals(“TABLET”)) {
} else {
public void tearDown() {
if (driver != null) {


Here, the methods are executed in the following order:


@BeforeClass methods will be executed once before all the methods, annotated as @Test

@Before methods will be executed before each of the methods, annotated as @Test

@Test method

@After methods will be executed after each of the methods, annotated as @Test

@AfterClass methods will be executed once after all the methods, annotated as @Test


Aside from annotations, that manage the order of test methods executions, there are annotations to parametrize tests and set up custom rules for additional flexibility.


Basically, JUnit will suit you in the following situations:


If you rely on a cross-platform solution.


If your testers have experience in Java.



nunit- test automation framework

NUnit is a port of JUnit to C# language. So, if you want a .NET solution, then you should consider it as one of the most viable choices. Basically, as a port, it allows you to write JUnit-style automated tests in C#, using similar techniques, such as annotations.




using System;
using NUnit.Framework;
using OpenQA.Selenium.Appium;
using OpenQA.Selenium.Appium.Android;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;
namespace Test_Selenium
class AndroidDemoTest
protected static readonly String ACCESS_KEY = Environment.GetEnvironmentVariable(“SEETEST_IO_ACCESS_KEY”);
protected static readonly String CLOUD_URL = “https://cloud.seetest.io:443/wd/hub”;
protected static readonly String TITLE = “Selenium/Appium Android test”;
protected DesiredCapabilities dc = new DesiredCapabilities();
private AppiumDriver driver;
protected String testName = “Selenium/Appium Android test”;
public void Initialize()
dc.SetCapability(“platformName”, “Android”);
dc.SetCapability(“app”, “com.example.tester.myapplication”);
dc.SetCapability(“appActivity”, “.MainActivity”);
dc.SetCapability(“appVersion”, “1.0”);
dc.SetCapability(“accessKey”, ACCESS_KEY);
driver = new AndroidDriver(new Uri(CLOUD_URL), dc);
public void AndroidNativeAppTestExample()
String text = driver.FindElement(By.XPath(“//*[@text=’Hello World!’]”)).GetAttribute(“text”);
Assert.AreEqual(text, “Hello World!”);
public void TearDown()
if (driver != null)


As you can see, NUnit is very similar to JUnit, it follows the same test design philosophy, utilizes a similar annotation mechanism. So, if you need a testing framework to organize automated testing in C#, then NUnit is the best choice.



testng- test automation framework

TestNG is, again, a new approach to JUnit. Like JUnit, TestNG utilizes Java programming language, inheriting all the benefits of Java cross-platform support.


What is the differences between TestNG and JUnit? Well… TestNG is striving to be a more configurable framework, allowing some additional grouping features as well as more hook annotations. For example, in TestNG, you can group your test classes to Groups and then group Groups to Suites. Besides, it supports annotations such as @BeforeGroup/@AfterGroup, @BeforeSuite/@AfterSuite, and others.


Choosing between JUnit and TestNG is not an easy task, so, usually, it is a matter of personal preference.



cucumber - test automation framework

Compared to all the previous examples, Cucumber is a different beast. Following aspects of the behavior-driven development paradigm, Cucumber describes high-level test scenarios in the Gherkin notation, very similar to a human language. For example:


Feature: Hello World.
Scenario: Click the button in my app
When I launch my app
And Save text before
And I click the button
Then I see something


However, when you have Gherkin scenarios, you need to implement the steps in a more low-level programming language. Cucumber supports steps definition in a lot of languages like Java, JavaScript, Ruby, Kotlin.


An example of step definition in Java:


import java.net.URL;
import java.util.concurrent.TimeUnit;
import cucumber.api.java.en.And;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.remote.AndroidMobileCapabilityType;
import io.appium.java_client.remote.MobileCapabilityType;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import static org.junit.Assert.assertEquals;
* Created by akaygorodova@issart.com on 03.02.2019.
public class MainActivitySteps {
private Logger logger;
private AndroidDriver driver;
private String text;
public MainActivitySteps() {
logger = LoggerFactory.getLogger(MainActivitySteps.class);
@When(“^I launch my app$”)
public void iLaunchMyApp() throws Throwable {
DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability(MobileCapabilityType.PLATFORM_NAME, “Android”);
caps.setCapability(MobileCapabilityType.DEVICE_NAME, “My device”);
caps.setCapability(CapabilityType.VERSION, “1.1”);
driver = new AndroidDriver(new URL(“”), caps);
driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
@And(“^Save text before$”)
public void readLabelTest() {
WebElement element = driver.findElement(By.id(“label”));
text = element.getAttribute(“text”);
logger.debug(“label/test: {}”, text);
assertEquals(text, “Hello World!”);
@And(“^I click the button$”)
public void iClickTheButton() {
@Then(“^I see something$”)
public void iSeeSomething() {
WebElement element = driver.findElement(By.id(“label”));
String actual = element.getAttribute(“text”);
logger.debug(“label/test: {}”, actual);
assertEquals(actual, text + “+”);


Besides that, you can use Cucumber in conjunction with JUnit, TestNG or NUnit framework to enhance your test cases with additional features.


So, you may consider Cucumber if:


You are more interested in high-level test scenarios.


You have a team with different programming experience, so you can distribute tasks of writing Gherkin test cases and implementation.


However, you should be aware that Cucumber is not designed for more low-level tests such as unit tests and API tests.


Experitest/SeeTest integration


All of the test automation framework from the examples above integrate with the SeeTest Platform in a simple manner. No matter what testing framework you have chosen, you can use Selenium and/or Appium drivers and specify access key and URL capabilities. Here is an example for Java programming language:


private static final String ACCESS_KEY = System.getenv(“SEETEST_IO_ACCESS_KEY”);
private static final String CLOUD_URL = “https://cloud.seetest.io:443/wd/hub”;
DesiredCapabilities dc = new DesiredCapabilities();
dc.setCapability(“accessKey”, ACCESS_KEY);
private AndroidDriver driver = new AndroidDriver(new URL(CLOUD_URL), dc);


In order to make this code work, you should add a new system environment variable “SEETEST_IO_ACCESS_KEY” and set its value to your SeeTest access key. Then you can refer to the driver variable to locate your device. This approach works with all the frameworks above, only programming languages may differ. When you run your tests, the driver will connect to the SeeTest cloud and execute your test.




So, to sum the things up:


If your preference is a Java programming language, then JUnit is a great choice for you. The main advantage of JUnit is its popularity.


If you are looking for a solution to work with the .NET framework and C# programming language, then Nunit is an obvious choice.


TestNG, being very similar to JUnit provides some extra features to organize your tests. It is slightly less widespread than JUnit, though.


Cucumber supports a great number of programming languages and offers a kind of unique ability to have test cases be described in Gherkin language and steps defined on the programming language of your choice. However, it seems to be not very convenient to use it for testing anything on the lower level than business/functional


Currently, there are enough test automation frameworks available for use. Some of them are very different from the others, some are similar. The main benefit of adopting one or another framework is that it will allow you to keep your tests uniformed, standardized and accessible.scenarios.


Konstantin Tonkov – Performance Engineer ISSArt QA team


Comments are closed.