Difference between revisions of "Firebase Test Lab"

Latest revision as of 17:55, 30 August 2017

Throughout programming history testing has been an integral part of the development process. With numerous combinations of device types and operating system versions, Android presents unique challenges in terms of app testing. It is no longer possible to test an app on a single device and Android version and assume that it will run with equal reliability on all other potential hardware and operating system combinations. Firebase Test Lab takes a large step towards addressing this problem by providing a testing platform that allows apps to be tested using many hundreds of device configurations simultaneously.

This chapter will introduce Firebase Test Lab and explain how it can be used to perform different types of testing. Subsequent chapters will cover the two main testing options, namely Robo testing and instrumentation testing, in greater detail.

An Overview of Firebase Test Lab

Firebase Test Lab allows you to define a collection of different device configurations (called a test matrix) on which a test is to be performed. Each individual device configuration (referred to as a test execution) is created by selecting from a range of device type, operating system version, language and orientation configuration options known as test dimensions.

All of the testing is performed using physical and virtual devices located at a Google data center allowing a wide range of device platforms to be tested without the need to purchase devices or configure multiple emulator sessions.

Once a test matrix has been defined, the APK file for the app is uploaded and testing started with a single button click. Once started, the tests in the matrix are performed simultaneously (up to a maximum of 200 device combinations). Once testing has completed, results of the test are provided including logcat output, screenshots, video playback and performance data. Firebase currently supports Robo, instrumentation and game loop testing.

Robo Testing

Robo testing allows apps to be tested without having to write any tests. To perform a Robo test, simply upload the APK file for the app and define the maximum duration of the tests and the depth within the app to which the test should be performed. Robo testing works by identifying the hierarchy of the user interface and then performing a sequence of simulated user actions (such as button presses, user input and screen navigation) intended to exercise as much of the app as possible. This testing will continue until either the app crashes or the maximum time is reached.

The main purpose of Robo testing is to quickly and easily find bugs that cause an app to crash. Robo test has no knowledge of the way the app is intended to work, so consequently has no way to verify that any response from the app to a particular action (aside from a crash) was the expected response. This level of testing requires the use of instrumentation testing. Robo testing with Firebase Test Lab is covered in the chapter entitled A Firebase Test Lab Robo Testing Example.

Instrumentation Testing

Instrumentation testing involves the creation of tests designed specifically to test an app and validate that the app behaves as expected in response to actions. A test could, for example, be written to click on a Button widget and then verify that the action caused specific text to appear on a TextView object elsewhere in the user interface layout.

Firebase Test Lab currently supports instrumentation tests created using the Espresso, Robotium and UI Automator Android testing frameworks.

Instrumentation testing with Firebase Test Lab is covered in detail in the chapter entitled A Firebase Test Lab Instrumentation Testing Example.

Game Loop Testing

Game loop testing is a new Firebase Lab Test feature that allows user actions to be simulated within game apps that use third-party user interface frameworks. Game loop testing involves the addition of code to the app being tested to simulate user interaction.

Test implementation is highly app specific in that test implementation beyond the basic setup steps will vary depending on the frameworks used and the game design and logic. More information on game loop testing can be found at:

https://firebase.google.com/docs/test-lab/game-loop

Test Lab Pricing

Firebase currently provides a choice of three Firebase plans named Spark, Flame and Blaze. At the Spark level testing is free but limited to 10 virtual device and 5 physical device test executions per day. When working through the tutorials in this book and experimenting with Test Lab, the Spark level should be more than adequate. Even though the Flame plan is a paid tier, it is subject to the same daily restrictions as the free Spark plan.

Unlimited testing is only available with the Blaze plan on which testing is charged at an hourly rate on a per device basis.

Summary

Firebase Test Lab allows Android apps to be tested on a wide range of device types and operating system versions simply by uploading an APK file to the Google cloud and designating the tests to be performed. Test Lab currently supports Robo, instrumentation and game loop testing.