Head of Sales
Why Mobile App Performance Testing is Important?
“What’s going on?”
The guys in quality assurance were frantic.
It was just yesterday that we celebrated the official release of our company’s mobile application. We tested all its functionality, sent it off to production, and for the first time in months, left at 5 PM.
Today was supposed to be our victory lap. We were planning a department lunch at one of the fancier steakhouses in the city. Our boss was going to expense it. Rumor had it the head of DevOps was going to surprise us with a bottle of fine wine.
When we walked in the office the next morning, nobody expected the carnage that would ensue.
“All mobile testers in my office, now. Not in 5 minutes, now, now NOW!”
The head of QA was in a fit. Beads of sweat poured down the brow of the head of DevOps, standing beside him.
“What the heck happened? The app doesn’t work. We are getting complaints all over the world from irate customers. The head of sales wants my head. Why is my application not working? Did you even bother to test it?”
Those were fighting words. Didn’t he see how many hours we put in, painstakingly going over every single function to see that it was working?
Jim, the team leader for mobile testing stepped in to defend us.
“I can assure you we did everything by the book. Each function was reviewed again, and again. You have all the documentation proving that we did our jobs right.”
“So why doesn’t the damn thing work?”
After a day of research, we found the problem. It was a performance issue. The mobile application does, in fact, work. But in some areas, it works very slowly. A page would download properly, but in 12 seconds. Within half that time the user was abandoning the site, and deleting the app. The user failed to distinguish between functional issues and performance issues.
Not that it matters. It’s not the job of the user to identify why he thinks the app is broken. If he thinks it doesn’t work, he will delete it nonetheless. One of the essentials of mobile app performance testing is that a mobile application can pass all of its functionality tests, but still have tons of flaws.
Key Benefits of Mobile Application Performance Testing: Finding the Bugs That Functional Testing Will Miss
One of the key challenges in mobile application testing comes when you conduct a thorough set of functional testing for your mobile application, and everything looks fine. If all the functions are working, the logic goes that there shouldn’t be any problems. This is where everything gets messy. In most cases, your mobile application testing tool is using a powerful, and direct connection between your app and the application server. A large enterprise can have a technical infrastructure worth millions, in some cases billions. This creates an artificial assumption that all of your users are as resourceful how their smartphones connect to your app.
Once people all over the world access your app, most from places with much weaker connectivity, it won’t work as fast as it did in the mobile testing lab.
Your mobile QA team did their job. But your customers are still unsatisfied. Why?
The simple answer is that the functions work, but they work very slowly. In the eyes of a user, they may as well be broken. One of the challenges in performance testing for a mobile application comes when everything looks perfect on a mobile testing checklist simply because the defects weren’t tested for.
Without adequate mobile performance testing for your mobile application, there will be a gap between the “clean bill of health” you get from your functional testing team, and the problems experienced by the customers who have to wait too long for a page to appear.
There are two primary reasons for the gap:
Undetected Performance Issues: When you do your testing on a server where things like CPU usage and battery drain are not simulated, the environment the customer uses your mobile application in real time is nothing like the environment you created for your functional testing. In a real user environment, there are hundreds of users who access the application server all at once. Unless you account for concurrent usage of your application, you cannot detect the performance issues your users may experience.
Network Virtualization: If I test my mobile application over a very fast WiFi network, it would be due to the fact that the business I work for invested a lot of money to make sure that its IT infrastructure is top of the line. That doesn’t mean the average user has the resources, or is in the location, or even physical position to guarantee the same level of connectivity. A WiFi network has high bandwidth and low latency. This is the ideal situation for any connection between your smartphone and the application server. Fast and smooth performance is a given. But every customer has a different environment. Their connection is usually one of low bandwidth, and high latency. A user can be close, or far away from a router, connection point, or even tower. He could be inside a bathroom, or even accessing the mobile application in the shower. This creates a huge amount of mobile performance testing challenges.
How Network Virtualization Works
The user sees the front end of the app. He selects the functions he wants to use, and expects certain events to occur in a timely manner.
What goes on underneath the hood is far different. Your smartphone is housing a mobile app that is constantly sending data back and forth from its application server. Mobile application performance boils down to how effectively the data moves back and forth from the phone to the server, and back.
There are two main factors that determine how the data traverses the journey to and from your smartphone: Throughput and Latency.
To explain these terms let’s use an example. Let’s assume that your application server is a water pumping station, and your phone is the kitchen sink. The water itself is the data that travels from the pumping station to the sink, or from the app server to your phone. Your network is the network of pipes the water has to traverse to get to you.
Throughput is the width of the pipe, or how much water can move in a second.
Latency is the length of the pipe, or how much time it will take for the water to travel the network of pipes from the pumping station to your sink.
The thicker the pipe, the more water can move through it. The shorter the path, the quicker the water can get there.
How the Smartphone Created Big Mobile Performance Testing Challenges
What are the main changes that happened once we moved from a desktop based internet to a mobile one?
Different Geographic Conditions: A personal computer is connected to a WiFi or LAN, and the connection is steady. Conditions are static. In the mobile age, the connection has become dynamic. One moment you can be inside a building, the next moment you can be outside of it. One moment you can be by a window, and the next moment you’re not. You can be in a moving elevator. You can be suddenly surrounded by other people and it will impact your network connection. There are so many things that effect your network environment in the mobile world, and they are constantly changing.
Built in Latency: When the internet goes global, the path gets longer. The pipes from the water station to the sink no longer travel from one end of the city to the other, they go to another state, country, even continent. Users in India can access your mobile application from a server in California. Users in Tokyo can receive data from a server in London. These situations create built-in latencies because data now has a longer journey. A packet of data can move across the world in 15 milliseconds on a fiber optic cable. It can travel almost at the speed of light, but it cannot go beyond that speed. This creates challenges for the performance of the mobile app in that the customer demands information in real time, but the application has to fetch that information from half a world away.
Built in Disconnectivity: At any moment a user can enter an elevator or go into a tunnel and experience a disconnection from the network environment.
Network Operators: Each operator has a different network. The internal effect of the operator may not be very high, but it does vary.
The Role of Network Virtualization
What are the differences between the lab environment that the mobile testing team works in and the real world conditions the everyday customer has to face?
If you understood the real world journey an application had to take to get from the server to your phone, you would be shocked that the system actually works!
In the lab, your devices are connected to a high-speed Wi-Fi connection. The device connects through the Wi-Fi to the application server. The path from the phone to the server is straight and simple: One route straight to the server, creating a very high throughput, and extremely low latency. Throughput is around 30,000 kilobytes per second, and latency is less than 2 milliseconds.
This eliminates performance issues altogether.
The real world conditions that the user has to face is a completely different reality.
Where the mobile network in the lab is as static as a desktop, your customers are using a slow, dynamic GSM network. The client’s mobile device is in one continent, and the application server is often in another. Instead of one Wi-Fi point with good bandwidth connecting the smartphone (or tablet) to the server, a data packet has to traverse Wi-Fi, antennas, routers, load balancers, and other equipment before it gets to its final destination.
In the real world, throughput is 50 kilobytes per second, or 6,000 times less than in the lab, and latency is 250 milliseconds, or over 125 times longer than the lab environment.
This is where functional testing falls short, and performance issues arise.
How This Effects Customer Experience
Your application assumes the data requested from the server will be delivered in a reasonable amount of time. If the data does not arrive in time, the application will alert the user to a timeout exception.
If a user enters a command, or requests information and the mobile application doesn’t react quick enough, the user will assume that the application is broken and probably delete it. In some cases, the user will alert his friends on social media that the entire application is broken – even users living in areas with connectivity that won’t produce these errors. Some customers will go so far as to stop buying anything with the brand name of the business that developed the app written on the product.
Performance is a vital component of application quality, brand reputation, sales, and the bottom line!
Network Virtualization Step by Step
Every business has the same problem:
How can I close the “performance gap” so my mobile application can appear as flawless in front of my customers as it does to my mobile testing team?
Enter the role of network virtualization. It is the process of simulating real-world network conditions in the lab so your mobile devices test each function according to the actual environment your users will face.
Experitest’s SeeTest Network Virtualization is the industry leading solution to realizing the benefits of network virtualization.
SeeTest NetworkVirtualization is very simple to deploy. All you need is an extra server and you are good to go. It works in conjunction with Experitest’s SeeTestAutomation and SeeTestCloud so you can conduct functional testing under real-world network conditions to determine not only if the function is working with the devices in your mobile testing coverage, but if these functions are working in a timely manner that meets the expectations of your end users.
SeeTest Network Virtualization enables you to pack in varied network conditions without having to go beyond your mobile testing lab. It is a software-only solution. You simply download it to a machine and you are set.
How Network Virtualization Works for Your Mobile Application Performance Testing
Step One: Define your profile.
At the initial step, SeeTest will prompt you to set the type of network conditions you want to test your application for both the mobile device the application is operating on, and the application server it will be communicating with.
For the device, you set the location (you have the option of setting it as US, Europe, or Asia), the operator, the type of connection (Wi-Fi, 3G, LTE), and the quality of the connection (good, medium, bad, dynamic). The dynamic option allows you to simulate different types of physical positions like riding through a tunnel, or entering an elevator.
For the server, you can asset a location (US, Europe, Asia) and the provider.
This is one of the primary benefits of network virtualization: Even as your device is in the lab, you can create a situation where the device is in Japan, the application server is in Manhattan, and the user is driving through the midtown tunnel on his way to Long Island.
Step Two: Manipulate the device to perform the test.
Using the reflection, which is a graphical representation of the device which appears on a computer screen, you can manipulate the smartphone itself from your desktop screen by using the keyboard and mouse.
You can apply the profile you made to the functional test, simulating the network conditions you created for the test to run under. Using the recorder, you can initiate a flow of steps that creates a test script to create the function that SeeTestAutomation will run as SeeTest Network Virtualization tracks the performance under your specified network conditions.
Step Three: See the Results
SeeTest Network Virtualization will indicate the profile and the device you just set up. You will see the bandwidth at the current moment being applied to the device as it runs throughout the test.
As you run your test, you will get the performance results and the functional.
Using a mobile lab, you can perform your mobile testing in parallel execution, testing the same function over a wide array of devices, tracking the performance of each function on every device. SeeTestCloud enables you to create a mobile lab within your company’s VPN, and Experitest’s SeeTestAutomation lets you run tests in parallel execution. You can cover a massive test area in both functional and performance testing in a reasonable amount of time to get your application to market sooner with no unpleasant surprises for your customers.
Conclusion: Mobile Application Performance Testing is the Key to Customer Satisfaction
The past Black Friday saw so much activity take place on mobile, that same store sales actually dropped from last year. The appearance, functionality, and performance of your mobile applications are rapidly becoming as important as the appearance, functionality, and performance of your physical location.
It is getting easier for a customer to move to a competitor if the user experience isn’t perfect. While functional testing is tested for pretty well, there is an unseen test coverage that many brands are missing, leading to bad customer experiences, and a lot of lost opportunities.
The best mobile application performance testing tools make sure that the results your mobile QA team sees will be exactly what your customers can expect from the quality of your mobile application, and the reliability of your brand.
Get Updates on Software Quality Assurance at
our Experitest LinkedIn Page