Difference between revisions of "A Firebase Test Lab Instrumentation Testing Example"

Revision as of 18:44, 28 August 2017

The previous chapters have introduced Firebase Test Lab and worked through an example focused on testing using Robo test. This chapter contains an example of instrumentation testing using the Espresso testing framework in conjunction with Android Studio. Topics covered include the creation of a test using Espresso and execution of that test from within both the Firebase console and remotely via Android Studio.

Since full coverage of Espresso testing is outside the scope of this book, this chapter will provide an introductory knowledge base from which to begin exploring other capabilities and features of the framework.

Getting the Test App

As with the previous chapter, the Database app will be used as the test subject in this tutorial. Follow the steps at the beginning of the previous chapter if you do not have a copy of this app already set up within Android Studio.

An Overview of Espresso

Espresso is a testing framework that allows automated instrumentation tests to be created that simulate user interaction within an Android app.

As described previously, Robo testing works by analyzing the user interface view hierarchy of an app and devising tests that exercise as much of the user interface as possible given time and depth restrictions selected by the user. Although Robo testing is ideal for quickly finding bugs within an app, it knows nothing about the logic of the app. Robo testing does not know, for example, that text needed to be added to a text field before a button is clicked. Nor does is it able to check an action caused the expected change elsewhere within the user interface. Robo test cannot, for example, check whether a click on a specific button caused the text displayed on a TextView widget to change.

Instrumentation testing frameworks such as Espresso define the actions that are to be simulated within the app, including the order in which they are to performed. Instrumentation testing also allows the results of an action to be verified ensuring, for example, that the correct text appears on the correct TextView in response to a click on a specific button. Since the tests are themselves programs, testing can include logic allowing different actions to be taken based on the results of individual test actions.

The tests are contained within an Espresso test class which is essentially a Java class containing the sequence of tests that are to be performed. The test class is bundled in a separate APK file and executed in the same process alongside the running app. The actions contained within the app are performed automatically and any discrepancies from the expected behavior reported to the tester.

Espresso test classes can be written by hand, or generated automatically using the Espresso Test Recorder built into Android Studio. A combination of these approaches is also common, involving a process of recording a test and then enhancing it by modifying the code.

Enabling Test Lab in the Project

Before implementing instrumentation testing within Android Studio, the app project must first be connected to a Firebase project. If the project is not yet connected, display the Firebase assistant panel within Android Studio using the Tools -> Firebase menu option and, within the assistant panel, unfold the Test Lab section and click on the Run Firebase Test Lab from Android Studio link. On the next screen, click on the Connect to Firebase button and select the Firebase Examples project.

Creating the Espresso Test Class

For this example, the Espresso test class will be created using the Espresso Test Recorder. From within Android Studio, select Run -> Record Espresso Test menu option and select a physical device or emulator session on which to run the app. The app will launch on the chosen deployment target and connect to the debugger. A window will also appear on the development system in which events will be displayed as they are recorded within the app.

[[File:]]

Figure 44‑1

Recording the Test

The objective of the test is to verify that a database record can be added, found and deleted within the app. To verify that these actions have been performed, assertions will be added to the test at key points to verify that expected information is displayed within the text fields of the user interface. With recording started, perform the following steps within the app:

1. Select the Product Name field and enter text which reads Firebase Book.

2. Select the Product Quantity field and enter 10.

3. Click on the Add button to add the record to the database.

4. Select the Product Name field and enter text which reads Firebase Book.

5. Click on the Find button.

At this stage a record has been added and then located using the Find button. Before moving to the next step, an assertion needs to be added to verify that the record was successfully found. This can be achieved by verifying that the value 10 is present in the quantity text field.

Assertions can be added during recording by clicking on the Add Assertion button located in the Android Studio Espresso recording dialog. Once clicked a screenshot of the app user interface will appear as shown in Figure 44‑2:

[[File:]]

Figure 44‑2

The first step in adding an assertion is to choose an element from the user interface. This can be achieved either by selecting the element within the screenshot, or using the Select an element from screenshot drop down menu. Select the quantity TextView object by clicking on it in the screenshot, then refer to the Edit assertion section of the dialog and verify that it is configured with a text is comparison with a value of 10:

[[File:]]

Figure 44‑3

Click on the Save Assertion button to add the assertion to the test class, then return to the running app and perform the following steps to verify that record deletion also works:

1. Click on the Delete button to remove the Firebase Books record from the database.

2. Select the Product ID field and enter text which reads Firebase Book.

3. Click on the Find button.

Assuming that the deletion was successful, the Product ID field should read No Match Found. Return to the recording dialog in Android Studio, click on the Add Assertion button and select the text view element to the right of Product ID in the screenshot. Verify that a test is assertion comparing against text that reads “No Match Found” has been configured before clicking on the Save Assertion button:

[[File:]]

Figure 44‑4

Complete the recording by clicking on the OK button in the recording dialog. When recording has finished a dialog will appear requesting a name for the Espresso test class file. Enter DatabaseActivityTest into the name field and click on OK.

Reviewing the Test Class

Once the test class has been saved it will be listed in the Project tool window under app -> java -> com.ebookfrenzy.database (androidTest) as shown in Figure 44‑5:

[[File:]]

Figure 44‑5

Locate this file, double-click on it to load it into the editor and review the code that makes up the test. The code for each action takes a similar format. The code to enter the book name into the product name field, for example, reads as follows:

ViewInteraction appCompatEditText = onView(
        allOf(withId(R.id.productName), isDisplayed()));
appCompatEditText.perform(replaceText("Firebase Book"), closeSoftKeyboard());
Similarly, the assertions also follow a standard pattern. The code fragment below, for example, checks that the quantity text field contains a string which reads “10”:
ViewInteraction editText = onView(
        allOf(withId(R.id.productQuantity), withText("10"),
                childAtPosition(
                        childAtPosition(                                  
                            IsInstanceOf.<View>instanceOf(
                              android.widget.TableLayout.class), 2), 1),
                                  isDisplayed()));

editText.check(matches(withText("10")));

Take some time to review the code to gain a familiarity with the way in which the Espresso framework API works.

Running the Test Locally

Before making use of Firebase Test Lab, it is worth taking time to make sure the test works correctly locally. To run the test on a local device or emulator session, right-click on the DatabaseActivityTest entry within the Project tool window and select the Run DatabaseActivityTest menu option. In the deployment target dialog, select a suitable device or emulator and click on the Ok button. The app will launch and run through the sequence of actions and assertion verifications. When the test completes, the results will appear in the Android Studio Run window (Figure 44‑6). Assuming the tests were recorded correctly, the window should indicate a successful test run.

[[File:]]

Figure 44‑6

Firebase Console Testing

Once the Espresso test class has been created it can be used to perform tests within the Firebase Test Lab. This can be initiated either using the Firebase console or from within Android Studio.

To run tests from within the Firebase console, open the console in a browser window, select the Firebase Examples project then click on Test Lab. In the Test Lab screen click on the Run a Test button and choose the Run an instrumentation test option.

An instrumentation test within the Test Lab requires both the standard and test APKs to be uploaded. In the Configure test screen (Figure 44‑7) click on the Browse button for the App APK to display the file selection dialog. Locate the folder containing the Database project, then navigate to the app/build/outputs/apk subfolder. Within the apk folder, select and open the app-debug.apk file.

Next, click on the Browse button for the Test APK, this time selecting the app-debug-androidTest.apk file.

[[File:]]

Figure 44‑7

When the files have uploaded, click on the Continue button and configure the test dimensions. For this example, deselect the default physical device and select the Nexus 9, API level 25 configuration from the Virtual devices section of the screen.

Initiate the test by clicking on the Start 1 Test button located at the bottom of the dimension configuration screen.

Wait for the test to complete, then click on the Nexus 9, Virtual, API Level 25 link in the matrix screen to view the results. Results screens are available for log output, video recording of the test and performance data. Playing back the video, you may find that the actual test actions occur too quickly to see if correct tests were performed. Ideally, some screenshots at key points during the test execution would be useful. While screenshots are taken automatically during Robo testing, code must be added to the test class file to initiate screenshots during instrumentation testing.

Adding the Screenshotter Library to the Project

Adding support for screenshots during instrumentation test runs is a multistep process that begins with the installation of a special library into the project. Begin by downloading this library from the following URL:

https://dl.google.com/firebase/testlab/cloudtestingscreenshotter_lib.aar

Once the library has been downloaded, it needs to be placed in a directory named aars located at the top level of the project file structure. Within Android Studio, switch the Project tool window into Project mode using the toolbar menu:

[[File:]]

Figure 44‑8

Within the project hierarchy, right-click on the Database entry and select the New -> Directory menu option. When prompted, name the new directory aars.

[[File:]]

Figure 44‑9

Using the file system browser for your operating system, locate the downloaded cloudtestingscreenshotter_lib.aar file and copy it. Return to Android Studio, right-click on the new aars directory in select the Paste menu option. With the library added, return the Project tool window to Android mode before proceeding.

Locate the build.gradle (Project: Database) file listed under Gradle Scripts in the Project tool window and edit it to add the aars directory to the repository declarations:

buildscript {
    repositories {
        jcenter()
        flatDir {
            dirs '../aars'
        }
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.3.2'
    }
}

allprojects {
    repositories {
        flatDir {
            dirs '../aars'
        }
        jcenter()
    }
}

repositories {
    jcenter()
    flatDir {
        dirs '../aars'
    }
}
task clean(type: Delete) {
    delete rootProject.buildDir
}

Next, edit the build.gradle (Module: app) file and add an entry for the library to the dependencies section:

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
.
.
    androidTestCompile (name:'cloudtestingscreenshotter_lib', ext:'aar')
.
.
}

The library will need access to external storage and the internet in order to function. The final task in adding the library to the project, therefore, is to add the following lines to the app -> manifests -> AndroidManifest.xml file as follows:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.ebookfrenzy.database">

    <uses-permission   
          android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.INTERNET"/>

    <application
.
.
</manifest>

Adding the Screenshotter Code

The next step is to add code within the test class at the points where a screenshot is needed. Edit the DatabaseActivityTest.java file and add screenshot code after the Add button and Delete buttons are clicked:

import com.google.android.libraries.cloudtesting.screenshots.ScreenShotter;
.
.
ViewInteraction appCompatButton = onView(
        allOf(withId(R.id.button3), withText("Add"), isDisplayed()));
appCompatButton.perform(click());

ScreenShotter.takeScreenshot("after_add", mActivityTestRule.getActivity());
.
.
ViewInteraction appCompatButton2 = onView(
        allOf(withId(R.id.button2), withText("Find"), isDisplayed()));
appCompatButton2.perform(click());
ScreenShotter.takeScreenshot("after_find", mActivityTestRule.getActivity());

Compile and run the test on a local device or emulator to verify that the test still runs, then execute another instrumentation test run within the Firebase console. When the tests are complete, the Screenshots tab will be present in the results and should contain the two screenshots taken during the test.

Running the Test from Android Studio

To start a Test Lab instrumentation test from within Android Studio, run DatabaseActivityTest once again. When the deployment target selection dialog appears, select the Cloud Testing tab as shown in Figure 44‑10:

[[File:]]

Figure 44‑10

The first step is to either choose or create a matrix configuration for the testing. Click on the button displaying the three dots to the right of the configuration selection menu to display the matrix configurations dialog:

[[File:]]

Figure 44‑11

To create a new configuration matrix, click on the plus button located above the list of configurations. A new unnamed configuration will appear beneath the Custom heading in the list (marked A in Figure 44‑12). Begin by entering Database App Test Configuration into the name field before working through the list of categories (B), in each case selecting one or more options from the selection panel (C).

[[File:]]

Figure 44‑12

For this example, make the following selections from the corresponding categories:

• Device – Virtual Nexus 9, HTC

• Platform – Android 7.1.x API Level 25 (Nougat)

• Locale – Select the default value for your Android Studio configuration

• Orientation – Portrait

Once the configuration is complete, click on the OK button to return to the Cloud Testing dialog. Before testing can begin, the test needs to be associated with a Firebase project. Within the Cloud Testing panel, click on the button to the right of the Please select a project… text and select the Firebase Examples project within the project selection dialog.

Verify that the matrix configuration and project settings are correct before clicking on the OK button to start the test execution. Android Studio will build the app and test APK files, upload them to the Test Lab and start the test. The status of the test will appear within the Android Studio Run tool window (Figure 44‑13) and may also be viewed within the Firebase console by clicking on the link provided in the Run window.

[[File:]]

Figure 44‑13

Summary

This chapter has provided an example of instrumentation testing using Firebase Test Lab and the Espresso Android testing framework. Espresso tests are contained in Java class files that run alongside the app during testing. These test classes can be written by hand or generated automatically using the Espresso Test Recorder built into Android Studio. Once the test has been created the APK files for both the app and the test class are uploaded to the Firebase Test Lab and used to run tests on selected device configurations. Firebase uploading and test initiation can be performed either within Android Studio or via the Firebase console. Unlike Robo testing, screenshots are not taken automatically when performing instrumentation testing. As demonstrated in this chapter, screenshots can be taken using an additional library and some extra code within the test class.