Selenium WebDriver is one of the most popular frameworks for web automation. To get more from your Selenium WebDriver tests there are many test frameworks, services, and libraries, which help you to organize your tests, monitor their execution, and provide comprehensive reports. This article describes solutions for three aspects in automation testing. Creation of parameterized tests, test execution reports, and logging for Selenium WebDriver.
Parameterized tests are necessary for cases when you need to execute the same suite of tests with different test data. You can write several identical tests, but with different input conditions, but it’s not an optimal solution. What you need is to add parameters for your tests. Thus, one test will be performed several times under different conditions.
Test execution reports help you to present test execution results in a friendly format. The TestNG framework offers a list of tests and statuses corresponding to them. Such statistics give a general idea of the success of the tests. Nevertheless, you may get intermediate results for each stage of tests execution into the reports. To do this, you need to add the Allure framework. It will show not only the statistics of test execution but also step-by-step actions in each test.
Logging in the Selenium WebDriver is a necessary thing for complex multi-step tests for tracking the logic of test case execution. For example, you can connect logging methods, implemented in Log4j library.
All the examples for this article are written in Java, use TestNG framework, Idea IDE and for some test examples, the Maven builder is used.
TestNG parameterized tests
How to launch a few tests with similar parameters
If some parameter is defined for all tests of a project.
Often in web automation, it happens that the same values have to be used in different tests. In this case, one can use the TestNG capacity to define a certain parameter for a test suite in the configuration file. For example, to provide the URL for the Selenium WebDriver method for execution of a test suite. The example of such a test written in TestNG framework is shown here.
Parameterized tests have to be launched with a builder, for instance, Maven. Now we are going to demonstrate how to feed parameters to the Selenium WebDriver tests with the TestNG test framework.
Selenium WebDriver tests with parameters
Passing simple values to a test.
If you need to create tests, which enable you to process data sets, you can use DataProvider. DataProvider is a method, which returns a two-dimensional array of objects. The first dimension of this array is the number of test launches. The second dimension of this array has to correspond to the number and types of the test parameters.
The method has annotation @DataProvider The example of code on simple test suites is shown here.
Let’s execute the command mvn clean test and check the execution results.
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.23 s – in TestSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
Passing iterative objects
The example with passing iterative objects is shown in the code below. In this example (found here) parameters are passed to the test method in the arraylist.
Data from a csv file.
The data required for the test execution can be written to a CSV file. In this case, the same iterative object can be used to pass data to the test. In the DataProvider method, data from the CSV file has to be copied to the ArrayList and then used in the method @Test. Data from the CSV file is read string by string and split by the delimiter char. For the code example let’s use the same data, as they are in the example below, but place them into the data.csv file. The changed code of the test will look in the following way.
data.csv structure is shown below:
Mobile Device & Browser Lab; Mobile Device & Browser Lab
Live Cross Browser Testing; Cross Browser Testing
Automated Mobile Testing; Mobile Test Automation
Data from a JSON file
Another format for the test data representation is the JSON format. The approach of feeding test data to tests is to use iterative objects. The only difference from the approaches shown above is that the DataProvider method has to process a JSON file and fill an ArrayList with data from it. For this task, we need to create a class in which JSON array elements will be passed. In the Selenium WebDriver test, we get values as these class instances. For the convenience of working with JSON objects, the library gson is used. The documentation for this library can be found here. Use this dependency section to add it to the maven project.
For the code example let’s use the same data, as they are in the example below, but place them into the data.JSON file. The changed code of the test will look in the following way.
Test report with Allure
After you have a considerable number of Selenium WebDriver tests you need to find an instrument to control their execution results. As a rule, you are interested in information about successful tests, and the number of tests whose execution finished with a failure. This basic information can be provided by the TestNG framework. It can be connected to your project from the Maven repository. Comprehensive documentation for the framework can be found at this link. After the tests are executed, an XML document with the execution results is formed and an HTML document for their presentation is created.
For example, we have three tests. Two of them are passed and one fails. After tests are executed the execution result will be printed out to the IDE console.
[INFO] Run 1: PASS
[INFO] Run 2: PASS
[ERROR] Run 3: TestNGReport.testLink:39 expected [Mobile Test Automation!] but found [Mobile Test Automation]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] BUILD FAILURE
TestNG also created a report and it can be found in /target/surefire-reports/index.html folder.
TestNG shows the list of tests which failed and their number, string in the test code where the error occurred and the error description. However, if we want to trace the tests execution step by step and get the statistics on test groups, statistics of tests execution, download the test execution results in CSV format, it’s recommended to connect your tests to the Allure framework and use it.
Allure – is a framework for displaying the test execution results and for generating test execution reports. If we use the maven builder, the Allure library has to be defined in the dependency section of the maven pom.xml file.
In the build section of the pom.xml file, we need to provide a path to the testng.xml file, which has the description of tests to be launched. In the file allure.properties we need to define a path to the folder with allure report as one of the parameters.
To demonstrate the displaying of a test execution steps, let’s create the test, using the data provider in which each step is described with the method @Step. Each test step description should be added in parameters. The code of the test is shown here.
Use mvn clean test command typed in the console to launch the test.
Test execution results will appear in the folder target/allure-results. To generate a report and open the Allure page, execute the command, typed in the console: allure serve target/allure-results. The main page of the Allure report shows the statistics of test executions, distribution tests over tests suites and categories.
You may navigate to the section with the list of tests. The statistics of the executed tests, each test execution time, list of parameters for each test, and test groupings are shown there as well.
Going to a specific test you may see a lot of information related to the test: A list of steps, the execution time of each step, status, category, list of parameters, error messages for a failed test and other information.
In this part, we discussed the basic set of Allure framework properties. Besides getting reports on the local environment, you may integrate it with Jenkins and use for continuous integration.
Generating log files in Selenium WebDriver tests
Let’s imagine that we have a complex set of Selenium WebDriver tests, and it’s necessary to track the status of both the tests themselves and the individual steps while the Webdriver is running. Perhaps we will need not only to display messages in the console but also to keep a full-fledged journal about what is happening in our test. To achieve these goals, we will use log4j. Below is an example of using the log4j library.
Create a set of primitive tests. To do this, create a class LogTest.
Launch these tests and you will see the following information in the IDE console:
As you can see, we used the standard display method – System.out.println(“”);,however, it is only suitable for printing out some information or sometimes for debugging tests and is not so convenient to use for keeping detailed logs. To implement real logging, use log4j library.
Apache Log4j library is a Java-based logging utility which allows you to write data to a file line by line. In addition, log4j has 7 logging levels that you can use for your needs, namely: ALL, DEBUG, INFO, WARN, ERROR, FATAL, TRACE, and also no-logging level – OFF.
Add log4j to your project:
Import the latest version of the library:
b) Add a reference to the library config file:
create a Logger instance, working with the LogTest.class before @BeforetTest annotation:
Logger log = Logger.getLogger(LogTest.class);
using the Configurator, specify the path to the future config file. To do this, add the following to the beginning of @BeforetTest annotation:
As you can see, the file is called log4j.properties and it is located in the folder of your project. It can be located in any directory, but the project folder is the most convenient one for this file.
c) Create the project config file:
The first that should be specified is the log level and a place where the log data will be printed out. You may specify the output direction for a specified log level.
[html_tags_show]log4j.rootLogger=ALL, file, stdout[/html_tags_show]
In the string above parameters: ALL – is a log level, file – which means that the logs will be printed out to a file, stdout – means that logs will be printed out to the IDE console.
You can find more info about configuration parameters here.
d) Update the code of tests, replacing the System.out.println() calls with calling log4j methods. Since it has already created an instance of the Logger log, you can use the following construction:
Launch the test and check the execution result in the IDE console:
and in the .log file:
Now you know how to connect and configure log4j logging for your Selenium tests. You can use different levels and amount of logging for various purposes, as well as output all logs or only critically important for you.
Now, let’s add special logging Exceptions, for the demonstration we will edit only one test – Experitest3:
Run the test, and see what we get in the end in the IDE console and in the log.
This example can be downloaded at the following link.
In this article we discussed only a few tips how to extend Selenium WebDriver capacity to get more from the web automation.
To learn how to integrate your test automation with SeeTest and Selenium WebDriver click here.