Reduce your Flutter CI cost easily!

CI costs can often go up quickly if you don't pay attention. Lately, the CI was beginning to cost too much to sustain a project.

With over 500 tests and many golden tests, each push would result in at least 12 minutes of CI, which is only for the testing part!

Moreover, the CI was running on macOS machines to match the goldens. That is something that could surprise most of you, but goldens are not equal on all platforms!

Let's be clear, there are similarities, but since macOS uses font smoothing, there is often a tiny difference between macOS and Linux. This means that we often choose to run the CI on macOS for the tests.

But as you can see, the CI is ten times more expensive for Linux machines.

So what can we do to reduce our CI cost? We decided to go back to Linux machines, but with a slight change.

Authorize a small difference in goldens

Our solution was to create a custom matcher (inspired by Cocoon).

const double _kGoldenDiffTolerance = 0.005;

class AppFileComparator extends LocalFileComparator {
  AppFileComparator(String testFile) : super(Uri.parse(testFile));

  @override
  Future<bool> compare(Uint8List imageBytes, Uri golden) async {
    final result = await GoldenFileComparator.compareLists(
      imageBytes,
      await getGoldenBytes(golden),
    );

    if (!result.passed && result.diffPercent > _kGoldenDiffTolerance) {
      final error = await generateFailureOutput(result, golden, basedir);
      throw FlutterError(error);
    }
    if (!result.passed) {
      log(
        'A tolerable difference of ${result.diffPercent * 100}% was found when '
        'comparing $golden.',
      );
    }
    return result.passed || result.diffPercent <= _kGoldenDiffTolerance;
  }
}

As you can see, this comparator is pretty simple to understand. We use the GoldenFileComparator result and add a small tolerance. If there is only a slight change between the two files, we can consider only the font smoothing has changed, and the test will pass.

To use this new file comparator, we will create a custom function that will act as a matcher.

Future<void> expectGoldenMatches(
  dynamic actual,
  String goldenFileKey, {
  String? reason,
  bool skip = false,
}) {
  final goldenPath = path.join('goldens', goldenFileKey);
  goldenFileComparator = AppFileComparator(
    path.join(
      (goldenFileComparator as LocalFileComparator).basedir.toString(),
      goldenFileKey,
    ),
  );

  return expectLater(
    actual,
    matchesGoldenFile(goldenPath),
    reason: reason,
  );
}

By using this function, you will create a new folder goldens in which you will save all your goldens.

You then need to change your call to

  await expectLater(
      find.byType(MyWidget),
      matchesGoldenFile('widget.png'),
  );

await expectGoldenMatches(
    find.byType(MyWidget),
    'widget.png',
);

And that's it! You now should be able to run your CI with Linux machines even if you're generating your goldens on macOS!

Of course, this method doesn't come without a tradeoff: you might make a small change to your design that will go unnoticed ?. But in my experience, this has not been the case for me on my project.