If the java default encoding is not set to utf-8, a handful of tests fails. This is likely because when opening files we do not specify what encoding to open them with, so it defaults to ISO-8859 or something and files containing UTF-8 get translated wrong.

A simple way to test this is to set the LANG environment variable to a non-utf8, e.g.

LANG="en_US" sbt test

That should result in a few errors because certain characters are converted to question marks, which results in actual results not matching expected results.

I'd guess the issue is that we aren't specifying an encoding when opening test files, and so we get random failures.