Difference between revisions of "A Firebase Test Lab Robo Testing Example"

Revision as of 18:07, 28 August 2017

Clearly, Firebase provides a number of different options when it comes to testing Android apps. In this chapter a sample app will be tested using Firebase Test Lab Robo testing. As will become clear in this chapter, while Robo testing provides a quick and easy way to perform very basic tests, complete and thorough testing is only possible using instrumentation testing.

Getting the Test App

For the purposes of this chapter, an example app named Database will be used as the test subject. The Android Studio project for this app can be downloaded as part of the code sample download available from the following web page:

http://www.ebookfrenzy.com/print/firebase_android

After opening the project in Android Studio, select the Tools -> Firebase menu option to display the Firebase assistant. Within the assistant panel, unfold the section and click on the Run Firebase Test Lab for Android from Android Studio link followed by the Connect to Firebase button. In the resulting dialog, connect the project to the Firebase Examples project.

Verify that the app is ready for testing by compiling and running it on a device or emulator session and adding, finding and then deleting a database record.

Running a Robo Test

To initiate a Robo test of the app, open a browser window and navigate to the Firebase console. Select the Firebase Examples project followed by the Test Lab link in the navigation panel.

Within the Test Lab screen, click on the Run a Test button and select Run a Robo test from the drop-down menu:

[[File:]]

Figure 43‑1

The first item needed by Firebase Test Lab is the APK file for the app to be tested. To upload the APK file, click on the Browse button in the configuration panel (Figure 43‑2) and traverse the local filesystem to the folder containing the Database project. Once located, navigate within the folder to the app/build/outputs/apk subfolder where a file named app-debug.apk should reside. Select this file and click on the Open button.

[[File:]]

Figure 43‑2

Once the APK file upload is complete, click on the Continue button to configure the test dimensions. It is, of course, possible to select multiple test devices (both physical and virtual) and combinations of API level, language and orientations to meet the testing needs of the app. Since this will likely exceed the limits of the free Spark plan level the default Nexus 5 API Level 23 will be suitable for this example:

[[File:]]

Figure 43‑3

Since the user interface for the app contains EditText widgets it would be useful to be able to have text entered into these fields during the test execution. This is possible using the Robo testing advanced options.

Turn on the Advanced switch and note that options are available to change the depth within the app and the maximum amount of time for which testing is to be performed. For this example, leave these values unchanged:

[[File:]]

Figure 43‑4

If an app requires user authentication, fields are also provided for entering user name and password information for a test account during the testing process. This consists of the resource names of the login and password EditText fields within the user interface of the app together with the values to be entered:

[[File:]]

Figure 43‑5

If the app uses Google Authentication it is not necessary to use these fields. Robo test will automatically create a temporary Google account and use it to sign into the app during testing.

An option is also available to specify the resource ID of any other input fields within the user interface together with a value to be entered. This is the option that will be used in this test. When the Database app was designed, the product name EditText widget was assigned an ID of productName which now needs to be referenced along with a product name to be entered during the test execution as illustrated in Figure 43‑6:

[[File:]]

Figure 43‑6

Leave the remaining default settings unchanged and enable the Save template option. Name the template “Nexus 5 Test” before clicking on the Start 1 Test button.

[[File:]]

Figure 43‑7

After starting the test, a new screen will appear (Figure 43‑8) indicating that the test is running at which point there is very little option, aside from cancelling the test, but to wait for the test to complete.

[[File:]]

Figure 43‑8

Reviewing Robo Test Results

Once the test has completed, the Firebase Test Lab screen will update to indicate whether the test passed, failed or returned an inconclusive result. A successful test run will look similar to the screen shown in Figure 43‑9 below:

[[File:]]

Figure 43‑9

To begin reviewing the tests results, click on the Nexus 5, API Level 23 entry in the results list.

The results screen contains a range of different screens containing information about the test execution. If issues were encountered during the test run, the Test Issues screen will provide a description of the issue.

The Logs screen will display logcat output captured during the testing process and is useful for identifying the cause of crashes or other failures. The log output is presented on multiple pages which can be navigated using the left and right arrows in the bottom right-hand corner of the screen. To access the entire log, click on the View Source Files button and select the logcat entry.

The output may also be filtered by level of severity using the menu located in the top left-hand corner of the screen:

[[File:]]

Figure 43‑10

The Screenshot screen contains screenshots of the app taken at various times during the test run. An inspection of these screenshots will probably show that Test Lab added a number of empty entries into the database during execution.

The Activity Map screen shows a visual representation of the path taken through the app during testing. Following the arrows on the map it will become clear that Robo test simply clicked on the buttons and selected items as they were added to the list.

To view a video of the test run, select the Videos tab and play back the video. In some cases the text may have been added to the database, while in others the Add button may have been clicked while the text field was empty. Robo testing is not a precision testing tool so this is to be expected. The system is simply trying out multiple combinations of actions in an attempt to make the app crash. A more controlled approach to testing will be performed in the chapter entitled A Firebase Test Lab Instrumentation Testing Example.

Finally, the Performance tab will provide information about the performance metrics of the app during the test execution including CPU, memory and Network usage (Figure 43‑11). Note that these metrics are only available when testing on a physical device:

[[File:]]

Figure 43‑11

Introducing a Bug

The final step in this example is to introduce a bug into the app to see if it is detected by Robo test. Return to the app in Android Studio, edit the DatabaseActivity.java file and comment out the newProduct() method as follows:

/*
public void newProduct (View view) {

    if ((quantityBox.getText().toString().isEmpty()) &&
            (productBox.getText().toString().isEmpty())) {
.
.
.
    }
}
*/

Compile and run the app on a device or simulator and check that a click on the Add button now causes the app to crash with an exception due to the missing method.

Within the Firebase console run another test, uploading the newly built APK file and using the Nexus 5 Test template saved earlier in the chapter. The Test Lab will report that the test has failed, and clicking on the test execution line item (Nexus 5, API Level 23) will display the Test Issues screen containing the exception message:

[[File:]]

Figure 43‑12

A review of the Activity Map screen will show that the last action performed was a click on the Add button.

Before continuing, be sure to uncomment the addProduct() method and rebuild the app so that the APK file no longer includes the bug.

Summary

This chapter demonstrated the use of Robo testing to perform automated testing of an Android app without the need to create specific tests. Robo testing performs a wide range of actions within an app by interacting with the user interface in as many ways as possible in an effort to identify areas where the app crashes. Running a Robo test involves uploading the APK file for the app, specifying the test dimensions and then running the tests. After the test completes a comprehensive set of results are available including log output, screenshots, video playback and performance metrics.