November 17, 2019 Jonny Steiner

How Experitest and AccessibilityOZ Incorporate Accessibility Testing into Mobile App Testing

Incorporate Accessibility and Performance into App Testing

Share this knowledge!

Accessibility is not only something that we at Experitest care about but something that Accessibility OZ cares about as well. That is why we joined forces to create this great overview of Accessibility testing as part of your mobile test automation. We also did a webinar together that you can watch here.

First a bit about Experitest. We provide digital assets like mobile devices, simulators, emulators, and browser testing. Experitest has around 1000 customers in many different industries like banking, telecommunication, airlines, automotive, and healthcare. We operate from eight global data centers located in the US, Canada, Germany UK in Israel, Singapore, Japan, and Australia. Experitest provides the infrastructure for executing tests and accessing devices for manual testing. We support open source frameworks like Appium, Selenium, XCUITest, and Espresso. Experitest hosts and manages the solution in one of our data centers. Some of our customers opt for the on-premise solution in their own facilities which we fully support. And as we mentioned in the topic of this post we allow for functional, performance and accessibility testing.

Now let’s take a look at Accessibility OZ. They have been operating for eight years and have offices in Australia, the USA, and Europe. 60% of their staff have some kind of significant disability and they offer three products OzArt, OzPlayer, and the OzWiki. Included in that is their automated accessibility testing tool. They are looking to build Experitest features into that product over the next 6-12 months.

For more be sure to check out their AccessibilityOZ database of accessibility issues. A lot of their collateral goes into depth as to why accessibility testing is so important. They also focus on what you need to test to make sure your web and mobile apps are accessible. This includes what developers need to improve accessibility on their projects.

Accessibility Issues and Legislation

The world of accessibility testing is led by common accessibility issues and governmental legislation. In the US and across the world in developed and nearly developed countries 20% of the population has a disability that affects their daily life. Worldwide regulations define online or digital accessibility as the ability for a person with a disability to understand and use a website application, mobile app, or program.

 The Department of Justice OCR, Section 508, Section 504 and the CVAA govern accessibility in the USA. In the EU, it’s governed by EN301549 in Australia, it’s governed by the Disability Discrimination Act. Globally if you want something to be accessible you need to follow the Web Content Accessibility Guidelines.

These regulations are important because they allow people with disabilities to access information like anybody else. If you want to follow the current US presidential campaign you can watch television or read articles on different native apps. 10 to 15 years ago that was not possible. It also allows people with disabilities to interact with others without being categorized as disabled. This is a really important requirement.

People often don’t want to disclose that they have a disability. And so, especially with the advent of social media, it is something that is really important to people with disabilities that they can actually participate without disclosing a disability that they might have. It also allows them to undertake activities that they’re not otherwise able to do. For example, going grocery shopping. 5-10 years ago someone with a significant disability would have a carer that would go shopping for them. Their carers were fairly limited in the options that they would be able to provide. Whereas with Whole Foods and Prime and similar services you can choose exactly what products you want down to the most minute details.

There are 5 types of disabilities

  • Vision-related
  • How the mind interprets information
  • Movement
  • Hearing
  • Mental health (not covered in WCAG)

If this is a topic that is interesting to you, take a look at the Accessibility OZ website where they discuss this in more detail.

Accessibility Legislation

Worldwide accessibility requires that companies conform to the laws and standards set by different global organizations. In the USA, for example, it is the Americans with Disabilities Act, which applies to all companies and organizations. This is equally true for Section 508 as well as section, 507. That legislation applies to companies selling to Federal State or Local Governments. In Australia, they have the Disability Discrimination Act which applies to all companies and organizations. In Europe, they have EN301549 which similarly applies to all companies and organizations.

While there are differences between each legislation what it does mean is that companies must make their digital content accessible and these laws have largely been in effect since 1990.

If your digital content does not meet these standards then you might be subject to legislation.

Accessibility Litigation

The first accessibility litigation took place in Australia in 1999 relating to the website of the Sydney Olympics. The website was sued for lacking accessibility and was then fined to the sum of twenty thousand dollars. This may not seem like a lot but when you factor in the $500,000 they had already invested in legal fees it is a pretty hefty sum.

In the USA in 2009, there was a very famous case in the USA where the retail chain Target was sued over their website not being accessible. Target ended up settling out of court for 6 million dollars, but not before they ponied up 3.5 million dollars in legal fees. That’s almost 10 million dollars because they had not executed any accessibility testing and their site was not accessible by blind people.

Also in the USA and also in 2009, there was a lawsuit against Netflix who argued that since they were only a website and not a brick and mortar store they were exempt from the Americans with Disabilities Act. The judge ruled against them to the tune of 750 thousand dollars and required Netflix to add closed captioning to their service within two years of the judgment.

There’s been quite a lot of litigation. Recently, the most famous example is Dominoes who were sued by a customer who was unable to order from their website even though he was using screen reading software. There’s been a lot of litigation and it has been increasing by quite significant percentages every year and should serve as a warning to companies whose websites are not accessible by everyone.

W3C Web Content Accessibility Guidelines (WCAG)

In order to make sure that your content is accessible, you need to follow the W3C written Web Content Accessibility Guidelines. The W3C is an international vendor-neutral organization. they released version one of their guidelines in 1999, Version 2 in 2008 and version 2.1 in 2018.

The Web Content Accessibility Guidelines (WCAG) were written by a panel of people that include accessibility specialists, people with disabilities and software vendors. In other words, people with the experience and knowledge of the life and needs of the disabled.

Native App Accessibility Testing Methodology

This has led to AccessibilityOZ creating a testing methodology in order to help with accessibility testing. They found that W3C version 2.1 left much out in terms of coverage for mobile. They also found that different companies worldwide had their own models of accessibility standards.

Since the latest accessibility guidelines had been written in 2008 it was time to amalgamate those guidelines and combine them with the standards that many companies have set internally.

When you take into account that the guidelines were written in 2008 you realize that these were written without any knowledge of smartphones. Now, in 2019 we have to also be able to provide accessibility for touch screens and screen orientation. For example, the W3C version 2 did require that everyone using a keyboard has access to every part of a site regardless of their ability. This did not include touch screens because they were not a thing yet, but now in version 2.1 that was released in 2018 does include touch screens as well as other criteria.

Touch target size, which is the size of a link on your mobile device is actually a triple a (AAA) requirement and these are the highest level of changes that need fixing.

The first thing is to define what to test. What we found when it comes to native apps is that a native app is much more streamlined in terms of the content that it provides. It’s really important to define the functionality of your native app. For example, an online banking site specifically is about transferring money or, paying bills. Whereas the banking website would have size store locations and working hours on how to apply for a home loan and features aimed at people who might not actually be current customers.

When it comes to native apps, It’s much more defined and so that you need to take that into account when you test. And it’s also really important to test common elements. This includes items like:

  • Navigation menu use
  • Headers and footers
  • Landing pages
  • Emergency alert pages
  • Login pages
  • Settings
  • Accounts
  • Profiles
  • Contact us pages
  • Real-time updates

It is also important to test processes such as:

  • Selecting a product
  • Adding to cart
  • Paying
  • Live chat
  • Help
  • Calendar or date
  • Geolocation or maps
  • High traffic areas

This is the breakdown of what you need to test. In terms of what you actually test, AccessibilityOZ broke it into three steps.

Step one is testing critical issues. What they found was that there are many traps. A trap according to accessibility is where a user is trapped within a component and cannot escape without closing the native app. There are many more traps in native apps than on desktop computers.

Whereas on the web using your desktop, you’re really only looking at one trap which is a keyboard trap where you people get trapped in a component, like a video player or something like that.

So an example of one of the traps that you need to look for is the screen reader’s flytrap so the screen reader user must always be able to activate an item on the current page or move back to the previous page and this applies to screen reader users.

Here is an example from Linkedin. They had actually fix this. On the LinkedIn app, you would type in a parsed search and then you get to a page that would say, do you want to share with all public connections. If you’re a touch user, you can select one of those options. If you were a screen reader user you couldn’t access any of those options and you couldn’t go back so your only option was to close the app and start again.

The second step is to test mobile-specific issues and we’ve broken mobile-specific issues into these categories:

  • Alternatives display
  • Actionable items
  • Navigational aids
  • Audio and video
  • Forms

So for example, and one that social media giant Instagram fails in is one of the display requirements which has an orientation. So in portrait orientation. It looks fine, but if you move the mobile to a landscape orientation, the orientation of the app does not change. And that’s an important requirement.

The third step is to test assistive technology and feature support, and this is why we are really excited about the growing relationship between AccessibilityOZ and Experitest. Other mobile app testing systems don’t have the ability to turn on assistive technologies and features, whereas Experitest has does provide that which is fantastic. Actionable items can be accessed and activated by the following assistive technologies or when the following feature is enabled.

All important content can be accessed by assistive technologies or when the following feature is enabled.

The features that you need to test on iOS are the voiceover keyboard, keyboard and switch zoom, and both color and grayscale mode. And on Android, you must test the TalkBack keyboard, keyboard, and switch magnification as well as color and grayscale and increase tech size.

If you are testing a Samsung device, you might need to test with a voice assistant, which is the Samsung screen reader. If you’re testing a Kindle, you might need to test the voice view, which is the Kindle screen reader.

Accessibility Testing with Experitest

Armed with that information what I would now like to show you is a brief demo of how you can create and run scenarios based around accessibility testing. In this example, we will focus on mobile.

Once you are in the Experitest Platform you can see the devices that are available. For the purposes of this example, we have opened an iOS device. This is a real device that is located in one of our Data Centers. At the top of the screen, we have two tabs one for automation and another for accessibility.

We are going to start with the automation tab because it will enable us to have a small insight into accessibility. We have an accessibility switch and it will show only elements within your user interface that have accessibility tags. This enables you to verify that you didn’t miss any element. It will also point out when you have an element that you shouldn’t. If you want to test accessibility elements of another app all you have to do is access the app and refresh the page while it is set to accessibility mode.

Now let’s take a look at the accessibility tab. We will do a quick run through you through some different accessibility scenarios. First of all, we have a colorblind view. So it will show you your application and you can use that to test the contrast of your text. There is also a blur option and a shrink option these filters can all be combined.

The next tool. I’m going to show you is as important as it is useful. We call it the UI inspector. This is a screen reader that acts as a voiceover. You can select the application and simply activate the UI inspector. We have three options two-finger swipe left and right, and a double-tap for perform. The navigation is very similar to how voice-over works on a screen reader. We use the navigation to verify that there are no traps and that the text that needs to be read is being read correctly.

And of course, if you would like to learn more about Experitest’s accessibility testing you can find it here.

Summing up Accessibility Testing

Accessibility Testing is clearly an important aspect of mobile app testing. Accessibility testing gives users the ability to use your app as if they were not disabled at all. This inclusion goes a long way to earning public trust and could even make a business look better. Another reason accessibility testing is so important is that it is the law. No one wants to alienate their users and also face litigation. So stay tuned for more from Experitest and AccessibilityOZ.

Accessibility testing is far from the first time we have combined technology with one of our partners. Have a look at this blog post for another example.

Share this knowledge!