Module ceylon.test (simple framework for writing repeatable tests) is an integral part of Ceylon SDK since its first version and in the latest release 1.2.1 brings several handy new features, namely:

• Parameterized tests
• Conditional execution
• Grouped assertions
• Tagging and filtering
• Extension points
• Reporting

Let's look at them in more details.

Parameterized tests

Parameterized tests allow developers to run the same test over and over again using different values, where each invocation of a test function is reported individually. A classical example for usage of parameterized tests is with a function computing Fibonacci numbers.

shared {[Integer, Integer]*} fibonnaciNumbers => 
    {[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 8] ...};

test
parameters(`value fibonnaciNumbers`)
shared void shouldCalculateFibonacciNumber(Integer input, Integer result) {
    assert(fibonacciNumber(input) == result);
}

In this example, we use annotation parameters to specify the source of argument values, which will be used during test execution. You can use any top level value or unary function with a compatible type as the source of argument values. The argument provider can be specified for the whole function, as in this example, or individually for each parameter, then the test framework will execute the test for each combination of provided values. For example, a function with one parameter whose argument provider yields two values and a second parameter whose argument provider yields three values, will be executed six times.

This functionality is based on a general mechanism, which can be easily extended, e.g. serving values from data file or randomized testing. For more information see documentation to ArgumentProvider and ArgumentListProvider.

Conditional execution

In some scenarios, the condition if the test can be reliable executed is known only in runtime. For this purpose it is useful to be able explicitly declare those assumption, as for example in following test. When the assumption is not met, verified with assumeTrue() function, then the test execution is interupted and the test is marked as aborted.

test
shared void shouldUseNetwork() {
    assumeTrue(isOnline);
    ...
}

Alternatively, it is possible to specify test condition declaratively, via custom annotation which satisfy SPI interface TestCondition. In fact the ignore annotation is just simple implementation of this concept.

Grouped assertions

Sometimes you don't want to interrupt your test after first failed assertions, because you are interested to know all possible failures. In that case you can use assertAll() function, which will verify all given assertions and any failures will report together.

assertAll([
    () => assertEquals(agent.id, "007"),
    () => assertEquals(agent.firstName, "James"),
    () => assertEquals(agent.lastName, "Bond")]);

Tagging and filtering

Test functions/methods and their containers (classes, packages) can be tagged, via annotation tag. For example, a test which is failing randomly for unknown reasons can be marked as unstable.

test
tag("unstable")
shared void shouldSucceedWithLittleLuck() { ... }

Those tags can later be used for filtering tests. Either in inclusive style (only tests with specified tag will be executed).

$ceylon test --tag=unstable com.acme.mymodule

Or visa versa for exclusion (only tests without specified tag will be executed).

$ceylon test --tag=!unstable com.acme.mymodule

Extension points

Extension points are general mechanisms which allow to extend or modify default framework behavior and better integration with 3rd party libraries (e.g. custom reporters, integration with DI frameworks). The easiest way to register extensions is with annotation testExtension, which can be placed on test itself, or on any of its container. Currently the following extension points are available, and new ones can be added if needed:

• listening events during test run TestListener
• creating test class instance TestInstanceProvider
• post processing of test class instance TestInstancePostProcessor
• describing parameterized test TestVariantProvider
• resolving all possible argument lists combination for parameterized test ArgumentListResolver

Reporting

These two last features have already been available for some time, but they could easily have slipped your attention. The first is nice html report with results of test execution, to enable it, run the test tool with --report option, it will be generated under report/test(-js) subdirectory.

The second is support for Test Anything Protocol (TAP), which allow integration with CI servers. To enable run the test tool with --tap option.

And if you don't have enough, just look on excellent library, built on top of ceylon.test which enables BDD style of test development and much more, called specks.

Ceylon has had support for unit testing since milestone four, but its functionality was pretty limited due lack of annotations and meta-model at that time.

Fortunately this is not true anymore! With version 1.0 of Ceylon we also released a completely rewritten ceylon.test module. So let’s see what’s new and how we can use it now.

Tests annotations

Tests are now declaratively marked with the test annotation and can be written as top-level functions or methods inside top-level class, in case you want to group multiple tests together.

Inside tests, assertions can be evaluated by using the language’s assert statement or with the various assert... functions, for example assertEquals, assertThatException etc.

class YodaTest() {

    test
    void shouldBeJedi() {
        assert(yoda is Jedi, 
               yoda.midichloriansCount > 1k);
    }

    ...
}

Common initialization logic, which is shared by several tests, can be placed into functions or methods and marked with the beforeTest or afterTest annotations. The test framework will invoke them automatically before or after each test in its scope. So top-level initialization functions will be invoked for each test in the same package, while initialization methods will be invoked for each test in the same class.

class DeathStarTest() {

    beforeTest
    void init() => station.chargeLasers();

    afterTest
    void dispose() => station.shutdownSystems();

    ...
}

Sometimes you want to temporarily disable a test or a group of tests. This can be done via the ignore annotation. This way the test will not be executed, but will be covered in the summary tests result. Ignore annotation can be used on test functions/methods, or on classes which contains tests, or even on packages or modules.

test
ignore("still not implemented")
void shouldBeFasterThanLight() {
}

All these features are of course supported in our Ceylon IDE. Where you can create a Ceylon Test Launch Configuration or easily select what you want to run in the Ceylon Explorer and in context menu select Run-As → Ceylon Test.

Test command

Our command line toolset has been enhanced by the new ceylon test command, which allows you to easily execute tests in specific modules.

The following command will execute every test in the com.acme.foo/1.0.0 module and will print a report about them to console.

$ ceylon test com.acme.foo/1.0.0

You can execute specific tests with the --test option, which takes a list of full-qualified declarations literals as values. The following examples show how to execute only the tests in a specified package, class or function.

$ ceylon test --test='package com.acme.foo.bar' com.acme.foo/1.0.0
$ ceylon test --test='class com.acme.foo.bar::Baz' com.acme.foo/1.0.0
$ ceylon test --test='function com.acme.foo.bar::baz' com.acme.foo/1.0.0

More details about this command can be found here.

Next version

In the next version, we will introduce other improvements.

There will be a test suite annotation, which allows you to combine several tests or test suites to run them together:

testSuite({`class YodaTest`,
           `class DarthVaderTest`,
           `function starOfDeathTestSuite`})
shared void starwarsTestSuite() {}

You will be able to declare custom test listeners, which will be notified during test execution:

testListeners({`class DependencyInjectionTestListener`,
               `class TransactionalTestListener`})
package com.acme;

And finally you will be able to specify custom implementation of the test executor, which is responsible for running tests:

testExecutor(`class ArquillianTestExecutor`)
package com.acme;

Please note, that these APIs are not final yet, and can change. If you want to share your thoughts about it, don't hesitate and contact us.

Today we would like to introduce a new eclipse plugin that allows you to run ceylon unit tests, and easily monitor their execution and their output.

It offers similar comfort and integration with IDE as JUnit and even more. Finally, it is part of the Ceylon IDE, so no additional installation is needed.

Getting started with your first ceylon unit test

The new test framework is in the Ceylon SDK module ceylon.test, though this is still an early version and contains only basic functionality, because we need annotations and meta-model support to make it really flamboyant.

So let’s start by importing the ceylon.test module in our module descriptor and writing our first test.

module com.acme.foo '1.0.0' {
    import ceylon.test '0.4';
}

import ceylon.test { ... }

void shouldAlwaysSuccess() {
    assertEquals(1, 1);
}

void shouldAlwaysFail() {
    fail("crash !!!");
}

These tests can be run like any ordinary ceylon application with the following code:

void run() {
    TestRunner testRunner = TestRunner();
    testRunner.addTestListener(PrintingTestListener());
    testRunner.addTest("com.acme.foo::shouldAlwaysSuccess", shouldAlwaysSuccess);
    testRunner.addTest("com.acme.foo::shouldAlwaysFail", shouldAlwaysFail);
    testRunner.run();
}

Which outputs:

======================== TESTS STARTED =======================
com.acme.foo::shouldAlwaysSuccess
com.acme.foo::shouldAlwaysFail
======================== TESTS RESULT ========================
run:     2
success: 1
failure: 1
error:   0
======================== TESTS FAILED ========================

Now let’s see the IDE integrations!

Launch configuration

Once you have created your tests, you can create a Ceylon Test Launch Configuration. In this dialog you can specify the tests to run, their arguments, run-time environment, etc…

In the launch configuration you can directly add test methods or whole containers, like classes, packages, modules, or entire projects.

As you would expect, a quick way to run ceylon tests is to right click in the Ceylon Explorer or in open editor and select Run-As → Ceylon Test.

Viewing test results

The Ceylon Test View shows you the tests run progress and their status. The toolbar has familiar functions:

• show next/previous failure,
• collapse/expand all nodes,
• filter to only show failing tests,
• scroll lock,
• buttons for relaunching or terminating,
• test runs history, and
• menu for view customisation (show elapsed time or show tests grouped by packages).

Comparing test results

In Test Runs History you can review test runs history and switch between runs.

But the real killer feature here happens if you open Compare Test Runs dialog, where you can see which tests have regressed, which tests have been fixed and which tests have been added or removed between two test runs, making it extra easy to get an idea of what your recent work did to the test suite.

Try it now

You can try it right away by downloading one of our Ceylon IDE development build.