The Functional Layer
Replacing ‘Duration’ with Speed Index
CPU and Battery
Conclusions and Recommendations
Some PRO Tips
The most important and obvious aspect of your testing plan is the functional layer. Your application should function as expected.
When addressing the functional aspect, it’s highly important to be able to provide quick feedback to the developers. How quick? Ideally – a few minutes. But 2 hours is also fine. The efficiency of the R&D organization is highly dependent on this immediate feedback.
The customer’s digital experience when using your mobile application has become a crucial factor for your business success. Even for an organization with various communication channels and assets – providing an excellent digital user experience is a necessity that grows day by day.
In this article, we will drill down into the building blocks of mobile application performance.
To better understand what impacts your mobile user experience performance we need to understand the concept of ‘Transaction’.
A ‘Transaction’ starts when the user clicks on an element in the mobile application. The click triggers a RestAPI call (or multiple calls) to the application backend server (services), the server responds with some information, the application receives and renders the response.
A transaction starts with some user interactions and ends when the application user interface has fully rendered the response.
Any banking application contains tens to hundreds of ‘Transactions’.
Now let’s understand transaction internals.
A transaction will start with user interactions on the mobile application, usually clicking on a button. For example, clicking on the ‘Login’ button.
This UI event will trigger a set of application logic that will result in several Rest API calls that will be executed in parallel or sequentially. Once the server will respond to the different API calls the application will render the results.
The following diagram shows a simplified view of low-level components in a transaction:
It starts with the user clicking on the application button.
Device Process – The application processes the event and triggers the RestAPI call.
DNS resolving – The lower level network stack resolves the URL into an IP address.
SSL handshake – The lower level network stack opens a connection to the server and performs the SSL handshake.
Send Request – The RestAPI request is sent (usually a GET or a POST request).
Get Response – The server analyzes the request, processes it and sends a response.
Render – The device analyses the response and renders it to the user.
As mentioned before, this is a simplified description of a transaction. A real case scenario is more complex, mainly around the amount of RestAPI calls that are needed.
The following diagram shows a more realistic scenario:
In this scenario the transaction triggers a RestAPI call (1), then 2 more RestAPI calls are triggered (2) and (3) (in parallel), and finally, once both calls finish, a 4th RestAPI call is triggered. The results are then rendered to the user.
So how does a transaction affect the mobile user experience?
Duration – Obviously we would like to keep the transaction duration as short as possible. Duration is the most popular metric (but is far from reflecting the user experience).
CPU and battery – We focus on mobile application performance. The CPU and battery consumption has a big impact on the user experience.
API call count – The number of RestAPI calls and the sequence of those calls (whether they are performed in parallel or in serial). The aim is to keep the number of API calls to a minimum and to run them in parallel.
Download data – In many cases we use RestAPI calls to download rich data like images and video. Reducing the amount of downloaded data will improve transaction performance.
Server response time – The time it takes the server to analyze the request and respond. When measuring on the client-side it will be easier to measure the ‘time for a first bit’.
Speed Index is a metric that can be used as an alternative to ‘Duration’. It is doing a better job of reflecting on the mobile user experience.
What exactly is a ‘Speed Index’?
The following example shows how can see 2 web pages load second by second over 5 seconds.
In the first example, the user sees nothing throughout the load time, and only in the 5th second, the entire page is loaded.
In the second example, after 2 seconds, the user is able to see the full frames, on the 3rd second most of the content is available and the user can start analyzing the page and finding the basic needs he has from that page.
The difference in the user experience between the 2 cases is huge. It emphasizes why you need an alternative for ‘Duration’. Speed Index takes this into account.
Usually, CPU and battery consumption go hand in hand – High CPU usage results in high battery consumption.
When looking into the transaction different phases:
Device Process – should be very short and shouldn’t consume significant CPU assets or battery.
RestAPI – can be long but the application CPU usage and battery consumption should be very close to 0.
Rendering – usually a short phase that shouldn’t impact the CPU and battery too much.
Keep the transaction duration as short as possible.
Minimize the number of API calls and in case of multiple API calls perform them in parallel.
Make sure to minimize the amount of data that the app downloads. We use this mainly when your application is downloading images, by scaling and compressing the image on the server-side.
Testing in bad network conditions – to get the best result you should perform all of the measurements in bad network conditions. For that, you will need to have a mobile environment that can simulate different network conditions like 3G and more.
My recommendation is to use 3G average (3G technology with an average reception signal) as the default network condition.
Create a baseline – in the beginning, it is hard to tell if 4 seconds and 20% CPU usage measured with your application is a good or a bad measurement. Create a baseline of data collected from previous versions of your application.
Then, if there are changes from your user performance baseline, you would like to get notified. For a more in-depth look at performance testing watch this podcast episode from Test Guild featuring Guy Arieli.