Mobile App QA: Doing Testing Right

53% of users will remove a mobile app once it has exposed its buggy side. The same reason can account for almost 40% of customers loss for a brand. It makes the companies pay more attention to their mobile app testing strategy. In this article, we’ve gathered some tips on mobile apps testing peculiarities and outlined the best practices one can conform to while testing your next mobile app with remote dedicated QA.

Mobile app quality: why does it matter?

According to the survey from Dimensional Research, users are highly intolerant of any software issues. As a result, they are quick to ditch mobile apps after just a couple of occurrences.

The key areas were mistakes are unforgivable are:

Speed:

• 61% of users expect apps to start in 4 seconds or less;
• 49% of users expect apps to respond in 2 seconds or less.

Responsiveness:

• 80% of users only attempt to use a problematic app three times or less;
• 53% of users uninstall or remove a mobile app with severe issues like crashes, freezes or errors;
• 36% of users stop using a mobile app if it is not battery-efficient.

Stability:

• 55% of users believe that the app itself is responsible for performance issues;
• 37% lose interest in a company’s brand because of crashes or errors.

The app markets, such as Google Play and App Store encourage users to leave reviews of apps. Low-point reviews will naturally lead to decreased app’s attractiveness.

‘Anyone can read your app store rating. There’s no way to hide poor quality in the world of mobile.’
Michael Croghan, Mobile Solutions Architect

‘Therefore,“metrics defining the mobile app user experience must be measured from the customer’s perspective and ensure it meets or exceeds expectations at all times.’
Dimensional Research

The findings reinforce the importance of delivering quality mobile apps. This, in turn, necessitates establishing proper mobile app testing procedures.

QA and testing: fundamentals

Quality assurance and testing are often treated as the same thing. The truth is, quality assurance is a much broader term than just testing.

Software Quality Assurance (SQA) consists of a means of monitoring the software engineering processes and methods used to ensure quality.

SQA encompasses the entire software development process. It includes procedures such as:

• requirements definition,
• software design,
• coding,
• source code control,
• code reviews,
• software configuration management,
• testing,
• release management,
• and product integration.

Testing, in its turn, is the execution of a system conducted to provide information about the quality of the software product or service under test. The purpose is to detect software bugs (errors or other flaws) and confirm that the product is ready for mass usage.

The quality management system usually complies with one or more standards, such as ISO 9000 or model such as CMMI.

Belitsoft leverages ISO 9001 certificate to continuously provide solutions that meet customer and regulatory requirements. Learn more about our testing services!

Mobile app testing: core specifics

• The mobile market is characterized by fierce competition and users expect app vendors to update their apps frequently. Developers and testers are pushed to release new functionality in a shorter time. It often results in a “fail fast” development approach, with quick fixes later on.
  
  
  

  Source:http://www.perfecto.io

  
  
• Mobile applications are targeted for a variety of gadgets that are manufactured by different companies (Apple, Samsung, Lenovo, Xiaomi, Sony, Nokia, etc.). Different devices run on different operating systems (Android, iOS, Windows). The more platforms and operating systems are supported, the more combinations one has to test. Moreover, OS vendors constantly push out updated software, which forces developers to respond to the changes.
• Mobile phones were once devised to receive and make calls, so an application should not block communication.
• Mobile devices are constantly searching for the network connection (2G, 3G, 4G, WiFi, etc.) and should work decently at different data rates.
• Modern smartphones enable input through multiple channels (voice, keyboard, gestures, etc.). Mobile apps should take advantage of these capabilities to increase the ease and comfort of use.
• Mobile apps can be developed as native, cross-platform, hybrid or web (progressive web apps). Understanding the application type can influence a set of features one would check when testing an app. For example, whether an app relies on internet connection and how its behavior changes when it is online and offline.

Mobile app testing: automated or manual?

The right answer is both manual and automated. Each type has its merits and shortcomings and is better suited for a certain set of tasks at the certain stages of an app’s lifecycle.

As the name implies, automated mobile app testing is performed with the help of automation tools that run prescripted test cases. The purpose of test automation is to make the testing process more simple and efficient. According to the World Quality Report, around 30% of testing is automated.

So where is automation an option?

• Regression testing. This type of testing is conducted to ensure that an application is fully functional after new changes were implemented. As regression tests can be repeated, automation enables to run them quickly. Writing test scripts will require some time initially. However, it will pay off with fast testing in the long run, as the testers will not have to start the test from scratch each time.
• Load and performance testing. Automated testing will do a good job when it is needed to simulate an app’s behavior strained with thousands of concurrent users.

• Unit testing. The aim of unit testing is to inspect the correctness of individual parts of code, typically with an automated test suite.

  ‘A good unit test suite augments the developer documentation for your app. This helps new developers come up to speed by describing the functionality of specific methods. When coupled with good code coverage, a unit test acts as a safeguard against regressions. Unit tests are important for anything that does not produce a UI.’
  Adrian Hall, AWS blog contributor

• Repetitive tasks. Automation can save the need to perform tedious tests manually. It makes the testing time-efficient and free of human errors.      

While the primary concern of automated testing is the functionality of an app, manual testing focuses on user experience.

Manual mobile app testing implies that testers manually execute test cases without any assistant automation tools. They play the role of end-user by checking the correct response of the application features as quickly as possible.

Manual testing is a more flexible approach and allows for a more natural simulation of user actions.

As a result, it is a good fit for agile environments, where time is extremely limited. As the mobile app unfolds, some features and functionality codes are also changing. Hence, automated test scripts have to be constantly reworked, which takes time. When working on a smaller product like MVP, manual testing allows to quickly validate whether the code behaves as it is intended.

Moreover, manual testing is a common practice in:

• Exploratory testing. During the exploratory testing, a tester follows the given script and identify issues found in the process.
• Usability testing. Personal experience is the best tool to assess if the app looks, feels and responds right. This facet is about aesthetics and needs a human eye. 

‘While automated tests can streamline most of the testing required to release software, manual testing is used by QA teams to fill in the gaps and ensure that the final product really works as intended by seeing how end users actually use an application.’
Brena Monteiro, Software Engineer at iMusics

Mobile app testing: where?

When testing a mobile app one typically has three options for the testing environment:

• real devices,
• emulators/simulators,
• a cloud platform.

Testing on real devices is naturally the most reliable approach that provides the highest accuracy of results.

Testing in natural conditions also provides an insight into how an app actually works with all the hardware and software specifics.

70% of failures occur because apps are incompatible with device OS versions, and customization of OS by many manufacturers.

About 30% of Android app failures stem from the incompatibility of apps with the hardware (memory, display, chips, sensors, etc.)

Such things as

• push-notifications,
• devices sensors,
• geolocation,
• battery consumption,
• network connectivity,
• incoming interruptions,
• random app closing

are easier to test on physical gadgets.

Perfect replication and bug fixing are also can be achieved only on real devices.

However, the number of mobile devices on the market makes it highly unlikely to test the software on all of them directly. The variety of manufacturers, platforms, operating systems versions, hardware and screen densities results in market fragmentation. 

Moreover, not only devices from different manufacturers can behave differently, but the devices from the same manufacturer too.

Source: mybroadband.co.za

Source:developer.android.com. The share of Android OS versions

When selecting a device’s stack, it is important not only to include the most popular of them but also to test an app on different screen sizes and OSes. Consumer trends may also vary depending on the geographical location of the target audience.

Source: https://www.kantar.com

As the names imply, emulators and simulators refer to special tools designed to imitate the behavior of real devices and operating systems.

An emulator is a full virtual machine version of a certain mobile device that runs on a PC. It duplicates the inner structure of a device and its original behavior. Google’s Android SDK provides an Android device emulator.

On the contrary, a simulator is a tool that duplicates only certain functionality of a device that does not simulate a real device’s hardware. Apple’s simulator for Xcode is an example.

‘Emulators and simulators “have many options for using different configurations, operating systems, and screen resolutions. This makes them the perfect tool for quick testing checks during a development workflow.’
John Wargo, Principal Program Manager for Visual Studio App Center at Microsoft

‘While this speeds up the testing process, it comes with a critical drawback — emulators can’t fully replicate device hardware. This makes it difficult to test against real-world scenarios using an emulator. Issues related to the kernel code, the amount of memory on a device, the Wi-Fi chip, and other device-specific features can’t be replicated on an emulator.’
Clinton Sprauve, Sauce Labs blog contributor

The advent of cloud-based testing made it possible to get web-based access to a large set of devices for testing mobile apps. It can help to get over the drawbacks of both real devices and emulators/simulators.

‘If you want to just focus on quality and releasing mobile apps to the market, and not deal with device management, let the cloud do it for you.’
Eran Kinsbruner, lead software evangelist at Perfecto

Amazon’s Device Farm, Google’s Firebase Test Lab, Microsoft's Xamarin Test Cloud, Kobiton, Perfecto, Sauce Labs are just some of the most popular services for cloud tests execution.

‘Emulators are good for user interface testing and initial quality assurance, but real devices are essential for performance testing, while device cloud testing is a good way to scale up the number of devices and operating systems.’
Will Kelly, a freelance technology writer

Mobile app testing: what to test?

Performance

Performance testing explores functional realm as well as the back-end services of an app. Most vital performance characteristics include energy consumption, the usage of GPS and other battery-killing features, network bandwidth usage, memory usage, as well as whether an app operates properly under excessive loads.

‘It is recommended to start every testing activity with a fully charged battery, and then note the battery state every 10 minutes in order to get an impression of battery drain. Also, test the mobile app with a remaining device battery charge of 10–15%, because most devices will enter a battery-safe mode, disabling some hardware features of the device. In this state, it is very likely to find bugs such as requiring a turned-off hardware feature (GPS, for example).’
Daniel Knott, a mobile expert

During the testing process, it is essential to check the app’s behavior when transiting to lower bandwidth networks (like EDGE) or unstable WiFi connections.

Functionality

Functional testing is used to ensure that the app is performing in the way in its expected. The requirements are usually predefined in specifications.

Mobile devices are shipped with specific hardware features like camera, storage, screen, microphone, etc., and sensors like geolocation, accelerometer, ambient light or touch sensors. All of them should be tried out in different settings and conditions.

‘For example, “every camera with a different lens and resolution will have an impact on picture dimension and size; it is important to test how the mobile app handles the different picture resolutions, sizes, and uploading photos to the server.’
Daniel Knott

No device is also safe from interruption scenarios like incoming calls, messages or other notifications. The aim is to spot potential hazards and unwanted issues that may arise in the event of an interruption.

One should not also forget that mobile apps are used by human beings who don’t always do the expected things. For example, what happens when a user randomly pokes at an application screen or inputs some illogical data? To test such scenarios, monkey testing tools are used.

Usability

The goal of usability testing is to ensure the experience users get meets their expectations.

Users easily get frustrated with their apps, and the most typical culprits on the usability side are:

• Layout and Design. User-friendly layout and design help to complete tasks easily. Therefore, mobile app testers should understand the guidelines each OS provides for their apps.
• Interaction. An application should feel natural and intuitive. Any confusion will eventually lead to the abandonment of an app.

However, the assessment of an app’s convenience by a dedicated group may be a bit subjective.

To get a more well-grounded insight into how your users perceive your app, one can implement A/B testing. The idea is to ship two different versions of an app to the same segment of end-users. By analyzing the users’ behavior, one can adjust the elements and features to the way the target audience likes it more. The practice can also guide marketers when making some strategic decisions.

Localization

When an app is targeted at the international market, it is likely to need the support of different languages to which devices are configured.

The most frequent challenges associated with localization mobile app testing are related to date, phone number formats, currency conversion, language direction, and text lengths, etc. What is more, the language may also influence a general layout of the screen.

For example, the look of the word “logout” varies considerably in different languages.

Source: http://www.informit.com

Therefore, it is important to think about language peculiarities in advance to make sure UI is adapted to handle different languages.

Final thoughts

The success of a mobile app largely depends on its quality.

‘The tolerance of the users is way lower than in the desktop era. The end-users who adopt mobile applications have high expectations with regards to quality, usability and, most importantly, performance.’
Eran Kinsbruner

Belitsoft is dedicated to providing effective and quality mobile app testing. We adhere to the best testing practices to make the process fast and cost-effective. Write to us to get a quote!