Integration testing in Flutter
Team Glean's Cameron gives us his take on a new approach to integration testing with Flutter5 min read Published: 19 Aug 2021
Integration tests are an important part of any software testing strategy. They provide stronger guarantees of correctness than unit tests, but at the cost of being harder to write and slower to run. However, using Flutter and the new
integration_test package lets us close the gap somewhat, while still giving us the same peace of mind that we expect from integration tests.
Migrating from 'flutter_driver'
When Flutter launched, integration tests were written using
flutter_driver, which allowed programmatic control of a Flutter app. When run with the
flutter drive command, Flutter would spawn 2 processes:
- A "driver" process on the host machine (i.e. the developer's laptop) that would send instructions to and receive data from the app
- The app itself, configured to listen to incoming connections from the driver process
This allowed the developer to write tests in plain Dart that could interact with the app. However, it had some major drawbacks:
- The driver code couldn't share any code with the app, or unit tests. In fact, because of the way it was compiled, it couldn't use any
- It relied on strongly-typed APIs, because of its inability to import
find.byType('EventsPage')is easy to mistype, and even easier to misread
- Any communication happened over the RPC channel between the 2 processes. If you wanted to read some internal state of the app, you would need to serialise a message and register a special handler in your app.
integration_test package was released to fix some of these issues. The main difference is that the code for tests runs in the same isolate as the app code itself (meaning it has access to the same memory). This essentially solves the issues listed above, as well as a few other nice benefits:
- Same API as component tests
- Able to share code with the app
- Internal app state is totally visible to tests,
runAppis called inside the test
- Since the tests are built into the app executable, they are now compatible with Firebase Test Lab, for running tests on physical devices
Before diving into the changes, a word on page objects.
Page objects are a simple abstraction that make it easier to read and write widget tests. Flutter tests give you very fine grained control over the lifecycle of the app, but sometimes this is overkill, and we just want a "sane default behaviour". For example:
However, since these depend on
flutter, we couldn't use them in driver tests, so for a while we maintained 2 sets of page objects. Not only that, but the underlying APIs (
flutter_driver) had quite a few subtle differences in behaviour that were unintuitive and error-prone.
Page objects vs. a real network
integration_test, we could use these page objects in our integration tests as well, but some modifications had to be made. Generally, it's common to use
tester.pumpAndSettle() in unit tests to "finish" your last input (e.g. a tap, a scroll, etc). It tells the test environment to:
- build and render a new frame
- check to see if new frames are scheduled
- repeat until there are no more frames scheduled
This gives a really nice behaviour in unit tests, since network calls are typically mocked and can resolve synchronously (or at the very least before the next frame). It's common for buttons to have subtle animations that may take 10-15 frames that wouldn't be caught by a single
In integration tests, however, if the animation finishes before the network request, the app will have reached stage 3 (no more frames scheduled), and
pumpAndSettle() will return, and your test will likely fail. To get around this, our page objects needed a little tweaking. Before migrating, when a page object "tapped" something it would call this:
wrapFinder is a utility method that improves ergonimics slightly. It just lets us write
page.tap('Event name') instead of
This naive approach was fine, as long as any network calls that needed to happen finished within the call to
pumpAndSettle. When this wasn't the case, we needed to wait for something to be visible:
Then when we define our page-specific methods, we can pass in a value to
waitFor when we know that we want to wait for something. Often, however, we follow a fairly standard pattern of:
button press -> wait for network request -> go to new page. For this default case, we can do better! There is a variant of
tapAndNavigate, which just performs a
tap, then returns a page object for a different page:
But since we know that, when changing to a new page, we usually want to wait for it to appear (and since Dart supports reified generics), we can simply pass
find.byType(S) in to make sure we wait into the new page has appeared.
This doesn't apply everywhere of course, and some methods needed manually overriding, but it helps to narrow the gap between integration tests and widget tests.
All in all, this makes our integration tests just as easy to write as widget tests. For example, here's a widget test for the login page (comments mine):
And now, the corresponding integration test:
Firebase Test Lab
All this is reason enough to use
integration_test, but there's an extra cherry on top.
Firebase Test Lab is a Google Cloud product that allows developers to submit Android or iOS native test binaries. As mentioned, this was previously impossible, because
flutter_driver required a Flutter-specific driver process running on the host machine, which was not supported.
But since all of the code is bundled into the native binary that Flutter builds for us (e.g.
libapp.so on Android), it behaves exactly like a regular native app.
We're still in early stages of exploring Firebase Test Lab, but initial impressions are promising. One aspect I haven't spoken about is performance.
Flutter apps can build in 3 modes,
release. There is a very large performance penalty imposed by
debug mode, but in exchange you get access to hot reload and devtools. When profiling, apps should be run in
profile mode (surprisingly!). However, when running on an emulator (e.g. in CI),
profile mode is disabled, because emulator performance isn't indicative anyway.
Firebase test lab uses physical devices, which lets us get over this hurdle and get accurate performance metrics for our builds, automated as part of a CI pipeline. We even get a video of the tests running, which can be very valuable when debugging a flake.
All in all, with well-organised unit tests, and a little time,
integration_test makes it easy to write high quality integration tests for Flutter apps.
Could you be our next dev?
At Team Glean, we're always on the lookout for talented people to join us.
To learn more about working with us, and to see the latest opportunities, follow the link below!
More from Tech BlogView All
Glean vs. the universe
In which developer Chris Moore persuasively argues that a recent Glean glitch was caused by… the sun?
Migrating a Flutter app to Null Safety
In this post, Team Glean’s Cameron talks us through the benefits and drawbacks of the ‘null safety’ upgrade to Dart.
How we discuss engineering improvements at Glean
Team Glean's Gwen talks us through Engineering's regular 'Entmoot' and how it could help your dev team work smarter