LogoRRRs Testing Automation

LogoRRR uses TestFX to perform integration tests of its user interface. This topic is very important to guarantee the correct behaviour of the application and is invaluable to detect regressions or subtle changes if new features are added.

There are many aspects to testing as I see it- One factor is the maintenance of such tests.

The main reason for creating tests is that your product profits from them!

If they don’t help you to move faster, or finish your project in time they are of no use. Of course, in product development they are invaluable and guarantee stability and are the only tool to discover regressions reliably.

For short lived projects tests may not be feasible - when code is really never touched again. You wouldn’t write tests for your prototype? For your demo for the next sales event? It all depends on your budget and your resources, sure.

For LogoRRR I decided it was time to pin down certain functionalities in integration tests, and invested quite some time to lay a foundation which gives me some confidence that the most important usecases work and survive new features, refactorings and ideas.

More is always better!?

Writing tests is an art in itself - they should be easy to understand, to write, to debug … The same concepts apply to production code, I do not differentiate between test and production code in fact. There have to be tools and abstractions also for tests.

Writing test suites for an existing application is tedious but always beneficial for projects code quality. LogoRRR got some new layers and APIs during the initial TestFX test development, simply because the code was used in new ways.

However, in the initial phase of a project, even more so in a agile project like LogoRRR, things can change quickly. Because applications need a certain time to stabilize on their APIs, UIs and behavior, it is important to strategically think about end2end tests. For example, writing a clickthrough of the most important workflows is a good investment. A common error is to specialize end2end tests on certain early versions of the user interface. Here it is very hard to keep tests and development in sync.

A common pitfall is to rewrite tests to follow the evolution of the user interface. This can be circumvented by a rigorous planning phase, but this is hard to pull of and needs an experienced development team, much perseverance and discipline.

On the other hand, after having reached a certain amount of complexity, application development profits from tests and progress stays on a linear path. To find the sweet spot between no tests, regressions and overengineered end2end which are written for a version of your application of yesteryear - that is not easy.

In the past 20+ Releases LogoRRR has become increasingly complex and the time was ripe for such tests, and TestFX is of great help here.

Implementation in LogoRRR

Again, Scala lends itself perfectly for writing concise tests - by using TestFX APIs one can create domain specific atomic actions which serve as building blocks for tests. This approach emerges naturally during test development if some attention is given to refactoring, code duplication and things like that.

@Test def openAndCloseTab(): Unit = {
    checkForEmptyTabPane()
    openFile(path)
    checkForNonEmptyTabPane()
    clickOn(lookup(UiNodes.LogFileHeaderTabs).query[StackPane](), MouseButton.SECONDARY)
    waitAndClickVisibleItem(CloseTabMenuItem.uiNode(fileId))
    checkForEmptyTabPane()
}

This is an example for a test, copied from a source code file - it is pretty readable I think.

Learnings

In order to get LogoRRR working with TestFX, some adjustment had to be taken care of in the code. For example, a new Service layer was introduced to abstract file handling or dealing with opening external applications (for dealing with urls for example). As expected, writing frontend tests yields new approaches and requirements for the code, which is a good thing in the long run. (Bugs were detected too 🤷🏼‍♂️)

One aspect which is often neglected in short running process development is the behavior when shutting down your application. More often than not resources aren’t properly cleaned up - why bother if the process will be killed anyway? This mindset leads to faulty code. Tests will expose such inconvenient truths very quickly … 💣

It is also hard to create tests which can be combined without interfering with each other. That means each test has to make sure its environment is properly setup prior to the execution. Afterwards it has to make sure to clean up and release all allocated resources.

TIP: Always have an eye on execution time, this accumulates very fast.

Finally, have a look at a video of the tests to get a taste on how such tests look like (and also how LogoRRR works in principle)

… an idea would be to use TestFX to create use case demos for documentation … 🧐

The header photo is by Sora Shimazaki, thanks very much :)