June 28, 2016 Experitest

Cognizant and Experitest Deliver Big on Papa John’s Mobile Testing

Share this knowledge!


Things are going well for Papa John’s. It all goes with being at the forefront of digital innovation. Being the official pizza sponsor of the national football league doesn’t hurt either.


In 2001, Papa John’s became the first national pizza chain to offer online ordering. Papa Johns is on the forefront of adapting new technologies. By 2010, they made over $2 billion in e-commerce sales and $1 million in mobile sales.


Today Papa Johns is the world’s third largest takeout and pizza delivery restaurant chain. It has 4,700 restaurants, spanning 37 countries and territories. 3,500 of these restaurants are located in the US.


That screams mobility! Pizza delivery is by far one of the ripest industries to benefit from developing the best mobile application. To give anyone the convenience to have a pizza brought to their door by merely pressing a button is the greatest improvement in creature comfort since the remote control!




The strategy to ramp up their mobile offerings secured Papa John’s at the #1 spot. It drove revenues to new heights. The risk was twofold: as more customers used the mobile application, just one bug could aggravate a huge amount of people. As technology powered forward, it was essential that Papa John’s not only developed new features to its mobile applications but tested and released them quickly and ahead of the competition.


Their regression testing alone took a week and a half. They knew that to be competitive they had to ramp it up.


So they sprung to action.


Papa John’s Calls for a Delivery of its Own



Papa John’s teamed up with Cognizant and Experitest to scale up their QA to meet their enhanced mobility needs.


Cognizant, as one of the leading system integrators worldwide, has hundreds of clients all with different needs for their mobile testing. Rather than use multiple tools to handle projects of different scopes, Cognizant utilized the scalability of Experitest tools to handle mobile testing challenges of any size. As a company grows, and it expands its mobile applications offerings, Experitest’s SeeTest Suite can expand to meet those needs, enabling the company to continue to grow.


The first challenge was to explain in in dollars and cents. QA does not bring in money. Its job is to reduce cost. The big question is – how well does a mobile application testing tool reduce costs relative to its investment? Are there any financial befits to the bottom line or even the sales line? What return on investment (ROI) will expanded mobile testing bring to Papa John’s?


Then there is the logistics. How exactly will Experitest and Cognizant bring these cost savings and increased revenue flows to the company? What exactly will it do?


The first argument made was that the partnership between Experitest and Cognizant brought instant economies of scale. The synergies of already existing collaboration between the two companies brought far greater value than what you saw in a power point presentation.


Experitest also reacts to needs very quickly. As technology changes at lightning speed, requirements change even within the time span of a project, demanding that the QA teams adapt. With top level customer service, tech support, and client relations Experitest would be able to support Papa John’s scale up with quick action help for any hiccups along the way.


The Experitest roadmap into DevOps and IoT evolution indicated that the company is well positioned to provide the best solutions to the next wave of application development.


The challenge: Enable Papa Johns to Automate their Mobile Testing




Papa John’s QA team had to objectives in its mobile testing. They needed to test both their native mobile app and their responsive website. They went to Cognizant as they upgraded their website to responsive. They needed something to help them with testing as far as operating systems, mobile devices.


The mobile application would be a bit more challenging.


The QA team knew that in expanding to mobile, they would be dealing with additional devices and mobile operating systems. They simply didn’t have the manpower available to handle the avalanche of testing scenarios they would have to cover in expanding their digital brand.


Like many companies experiencing mobile for the first time, the new level of test coverage is a steep increase.


At the time, Papa John’s had limited resources in manpower as well as available devices they could test on. Cognizant’s solution was to upgrade Papa John’s QA to automation for both iOS and Android. They chose Experitest’s SeeTestAutomation for the task. Cognizant also selected Experitest to set up a dedicated mobile device lab in their Phoenix office.


Cognizant and Experitest Deliver Big on Papa John’s Mobile Testing

Papa John’s QA team gave Cognizant a list of the devices they needed in the mobile lab. The new mobile lab gave all of Papa John’s testers in their Kentucky and India offices total access to all the devices in the lab.


This addressed the main needs of Papa John’s QA team:


  1. Cross platform website testing: making sure that the same automation test script would run on Android and iOS.
  2. Making sure the same scripts for their web applications would run on Chrome and Safari.
  3. Enabling parallel execution of tests over multiple devices to reduce total testing time significantly.


Cognizant’s automation team created scripts for SeeTest to handle. Experitest engineers worked closely with Papa John’s team to make sure these scripts were operating on SeeTest smoothly.


SeeTest Delivers in 30 Minutes or Less

Cognizant and Experitest Deliver Big on Papa John’s Mobile Testing

The advantage of Experitest’s cloud tool was that once Papa John’s QA team completed its work in North America, Cognizant’s QA team would begin work on the same devices from their offices in India. Cognizant and Experitest scaled Papa John’s QA operations up to a 24/7 continuous testing environment.


As a result, Papa John’s enjoyed the following improvements to its productivity:


    • They had more people working on the same project.


    • More devices were available for parallel execution.


    • Using SeeTestAutomation, they could run automated test scripts over a wide array of devices, operating systems, and browsers all at once.


    • Once the QA team at Papa John’s walked into the office in the morning, they had reports from the results of the work done by both teams.


  • Their mobile testing ops were executing nonstop until each new version was released.


The biggest challenge for everyone was adapting to new technology. If a new release of iOS 9 comes out 3 hours before a test script is set to run, the testing tool must be able to immediately react to the new version and test for it. The extensibility of SeeTest allowed for in-project surprises to be scaled with minimal to no disruption.


Cognizant and Experitest Deliver Big on Papa John’s Mobile Testing

Cloud Solutions Guarantee Tight Security for Mobile Apps


A huge concern for Papa John’s was security. Their testing team is located in Kentucky, and they will be accessing a pool of devices located 2,000 miles away in Phoenix. At the same time, a team of testers from India will be accessing the same set of devices from over 7,500 miles away. They needed to know that the code they were putting on the devices would be secured from outside influences.


Experitest was able to accomplish this by working within the VPN of Papa John’s IT infrastructure.


Another big security must for Papa John’s was that all of the devices they were using were not shared, but dedicated solely to their company.


Cognizant and Experitest Deliver Big on Papa John’s Mobile Testing

The key for a multinational organization such as Papa John’s was to simulate operating under different mobile networks. The mobile device lab enabled different phones to simulate conditions from different mobile operators utilizing SeeTest Network Virtualization. This tool also enables the testing team to simulate any local mobile environment, even what is going on when a user takes his phone through a tunnel or on an elevator.


Papa Johns’ ROI from Mobile Testing


Originally Papa John’s was conducting its regression testing manually. It would take a week and a half to complete the task. Using SeeTestAutomation (https://experitest.com/test-automation-tool-for-mobile-testing-continuous-integration/) they were able to automate the process by writing 200 automated scripts with help from Experitest’s development support team. The process now takes 17 hours. Papa John’s reached a whopping 90% test coverage for their native apps. Papa John’s is now able to bring the most up to date version of its mobile application to market sooner than ever.


Cognizant and Experitest Deliver Big on Papa John’s Mobile Testing

Originally Papa John’s had one tester running all the scripts manually. By making use of a dedicated mobile device lab, they were able to run tests in parallel execution and scale up their mobile testing exponentially. As an Agile company, Papa John’s had the flexibility to make these changes quickly.


These improvements went beyond the QA team, and even the entire IT department. Word of the increase in productivity spread so fast that other departments were making strategic decisions based on the faster turnaround.


They were also able to find defects a lot faster, giving the development more time to fix them. When a QA engineer found a specific device running on a certain version of a specific mobile operating system and saw that a particular function was misfiring, they were able to report that defect to the developer.


Cognizant and Experitest Deliver Big on Papa John’s Mobile Testing

With the same access to the dedicated mobile lab, the developer didn’t have to reproduce the test case. He didn’t have to worry about finding another device just like the one the tester reported, with the same mobile OS, and version. By the click of a button, he could access the same device and immediately re-create the same case. This saved the developer lots of time, and the company lots of money.


From the time their new mobile QA resources were set up, Papa John’s recouped their entire investment within a handful of release cycles.


For continuous integration Papa John’s uses Jenkins. Jenkins is a CI platform which integrates into SeeTest seamlessly so Papa John’s is able to integrate its automated mobile testing into the rest of it’s automated DevOps infrastructure, shortening it’s development to deployment cycle even further.


mobile enterprise apps one size fits all


Dana Natan


Head of Marketing


Share this knowledge!