Module ceylon.test (simple framework for writing repeatable tests) is an integral part of Ceylon SDK since its first version and in the latest release 1.2.1 brings several handy new features, namely:

• Parameterized tests
• Conditional execution
• Grouped assertions
• Tagging and filtering
• Extension points
• Reporting

Let's look at them in more details.

Parameterized tests

Parameterized tests allow developers to run the same test over and over again using different values, where each invocation of a test function is reported individually. A classical example for usage of parameterized tests is with a function computing Fibonacci numbers.

shared {[Integer, Integer]*} fibonnaciNumbers => 
    {[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 8] ...};

test
parameters(`value fibonnaciNumbers`)
shared void shouldCalculateFibonacciNumber(Integer input, Integer result) {
    assert(fibonacciNumber(input) == result);
}

In this example, we use annotation parameters to specify the source of argument values, which will be used during test execution. You can use any top level value or unary function with a compatible type as the source of argument values. The argument provider can be specified for the whole function, as in this example, or individually for each parameter, then the test framework will execute the test for each combination of provided values. For example, a function with one parameter whose argument provider yields two values and a second parameter whose argument provider yields three values, will be executed six times.

This functionality is based on a general mechanism, which can be easily extended, e.g. serving values from data file or randomized testing. For more information see documentation to ArgumentProvider and ArgumentListProvider.

Conditional execution

In some scenarios, the condition if the test can be reliable executed is known only in runtime. For this purpose it is useful to be able explicitly declare those assumption, as for example in following test. When the assumption is not met, verified with assumeTrue() function, then the test execution is interupted and the test is marked as aborted.

test
shared void shouldUseNetwork() {
    assumeTrue(isOnline);
    ...
}

Alternatively, it is possible to specify test condition declaratively, via custom annotation which satisfy SPI interface TestCondition. In fact the ignore annotation is just simple implementation of this concept.

Grouped assertions

Sometimes you don't want to interrupt your test after first failed assertions, because you are interested to know all possible failures. In that case you can use assertAll() function, which will verify all given assertions and any failures will report together.

assertAll([
    () => assertEquals(agent.id, "007"),
    () => assertEquals(agent.firstName, "James"),
    () => assertEquals(agent.lastName, "Bond")]);

Tagging and filtering

Test functions/methods and their containers (classes, packages) can be tagged, via annotation tag. For example, a test which is failing randomly for unknown reasons can be marked as unstable.

test
tag("unstable")
shared void shouldSucceedWithLittleLuck() { ... }

Those tags can later be used for filtering tests. Either in inclusive style (only tests with specified tag will be executed).

$ceylon test --tag=unstable com.acme.mymodule

Or visa versa for exclusion (only tests without specified tag will be executed).

$ceylon test --tag=!unstable com.acme.mymodule

Extension points

Extension points are general mechanisms which allow to extend or modify default framework behavior and better integration with 3rd party libraries (e.g. custom reporters, integration with DI frameworks). The easiest way to register extensions is with annotation testExtension, which can be placed on test itself, or on any of its container. Currently the following extension points are available, and new ones can be added if needed:

• listening events during test run TestListener
• creating test class instance TestInstanceProvider
• post processing of test class instance TestInstancePostProcessor
• describing parameterized test TestVariantProvider
• resolving all possible argument lists combination for parameterized test ArgumentListResolver

Reporting

These two last features have already been available for some time, but they could easily have slipped your attention. The first is nice html report with results of test execution, to enable it, run the test tool with --report option, it will be generated under report/test(-js) subdirectory.

The second is support for Test Anything Protocol (TAP), which allow integration with CI servers. To enable run the test tool with --tap option.

And if you don't have enough, just look on excellent library, built on top of ceylon.test which enables BDD style of test development and much more, called specks.

With Ceylon we try our best to make every developer’s life easier. We do this with a great language, a powerful IDE, a wonderful online module repository, but also with an amazing command-line interface (CLI).

Our command line is built around the idea of discoverability where you get a single executable called ceylon and lots of subcommands that you can discover via --help or completion. We have a number of predefined subcommands, but every so often, we want to be able to write new subcommands.

For example, I want to be able to invoke both Java and JavaScript compilers and generate the API documentation in a single command ceylon all, or I want to be able to invoke the ceylon.formatter module with ceylon format instead of ceylon run ceylon.formatter.

Well, with Ceylon 1.1 we now support custom subcommands, fashioned after the git plugin system. They’re easy to write: just place them in script/your/module/ceylon-foo and package them with ceylon plugin pack your.module, and you can publish them to Herd.

Now every one can install your CLI plugin with ceylon plugin install your.module/1.0 and call them with ceylon foo.

What’s even better is that they will be listed in the ceylon --help and even work with autocompletion.

ceylon.formatter uses one, and I encourage you to install them with ceylon plugin install ceylon.formatter/1.1.0 and format your code at will with ceylon format :)

ceylon.build.engine also defines one and it just feels great being able to build your Ceylon project with ceylon build compile, I have to say. Although, unfortunately that particular module has not yet been published to Herd yet, but hopefully it will be pushed soon.

You can find out all about them in our reference.

Ceylon has had support for unit testing since milestone four, but its functionality was pretty limited due lack of annotations and meta-model at that time.

Fortunately this is not true anymore! With version 1.0 of Ceylon we also released a completely rewritten ceylon.test module. So let’s see what’s new and how we can use it now.

Tests annotations

Tests are now declaratively marked with the test annotation and can be written as top-level functions or methods inside top-level class, in case you want to group multiple tests together.

Inside tests, assertions can be evaluated by using the language’s assert statement or with the various assert... functions, for example assertEquals, assertThatException etc.

class YodaTest() {

    test
    void shouldBeJedi() {
        assert(yoda is Jedi, 
               yoda.midichloriansCount > 1k);
    }

    ...
}

Common initialization logic, which is shared by several tests, can be placed into functions or methods and marked with the beforeTest or afterTest annotations. The test framework will invoke them automatically before or after each test in its scope. So top-level initialization functions will be invoked for each test in the same package, while initialization methods will be invoked for each test in the same class.

class DeathStarTest() {

    beforeTest
    void init() => station.chargeLasers();

    afterTest
    void dispose() => station.shutdownSystems();

    ...
}

Sometimes you want to temporarily disable a test or a group of tests. This can be done via the ignore annotation. This way the test will not be executed, but will be covered in the summary tests result. Ignore annotation can be used on test functions/methods, or on classes which contains tests, or even on packages or modules.

test
ignore("still not implemented")
void shouldBeFasterThanLight() {
}

All these features are of course supported in our Ceylon IDE. Where you can create a Ceylon Test Launch Configuration or easily select what you want to run in the Ceylon Explorer and in context menu select Run-As → Ceylon Test.

Test command

Our command line toolset has been enhanced by the new ceylon test command, which allows you to easily execute tests in specific modules.

The following command will execute every test in the com.acme.foo/1.0.0 module and will print a report about them to console.

$ ceylon test com.acme.foo/1.0.0

You can execute specific tests with the --test option, which takes a list of full-qualified declarations literals as values. The following examples show how to execute only the tests in a specified package, class or function.

$ ceylon test --test='package com.acme.foo.bar' com.acme.foo/1.0.0
$ ceylon test --test='class com.acme.foo.bar::Baz' com.acme.foo/1.0.0
$ ceylon test --test='function com.acme.foo.bar::baz' com.acme.foo/1.0.0

More details about this command can be found here.

Next version

In the next version, we will introduce other improvements.

There will be a test suite annotation, which allows you to combine several tests or test suites to run them together:

testSuite({`class YodaTest`,
           `class DarthVaderTest`,
           `function starOfDeathTestSuite`})
shared void starwarsTestSuite() {}

You will be able to declare custom test listeners, which will be notified during test execution:

testListeners({`class DependencyInjectionTestListener`,
               `class TransactionalTestListener`})
package com.acme;

And finally you will be able to specify custom implementation of the test executor, which is responsible for running tests:

testExecutor(`class ArquillianTestExecutor`)
package com.acme;

Please note, that these APIs are not final yet, and can change. If you want to share your thoughts about it, don't hesitate and contact us.

About the Ceylon command-line interface

We’re programmers, so let’s face it, we spend lots of time in those small obscure windows with tiny text, preferably with a green or black background, that Hollywood movies often display scrolling a lot, really fast, you know? Terminals.

While good IDEs are required, and the Ceylon IDE is a must-have for Ceylon developers, they can never replace a good terminal session with the proper command-line interface (CLI) tools.

Oh, I know Eclipse has good Git support and I know a few colleagues who use it for committing and branching and other daily, nay, hourly push/pull command. But I really love my CLI when it comes to using Git, I never use Eclipse for that, so I spend a lot of time in my terminal.

Also, it’s essential to be able to build a Ceylon project from the command-line, and not just from the IDE. Imagine trying to build your project in a remote machine using an IDE with a remote X session? No, that doesn’t cut it, sorry.

So from the start, Ceylon had a CLI. Actually, we had a CLI before we had an IDE, but that didn’t last long. And in the beginning we copied Java and had a few commands such as ceylonc to compile, and ceylon to run. Then we added the API generator tool called ceylondoc, and very soon after that we figured we were doing something horribly wrong.

Back when Java came out in 1995 it was sorta cute and useful that all its commands started with the java prefix, but after you got around to remembering what javac and javadoc stood for, you started to wonder what the hell javap or javah could possibly mean.

Then Git came around and revolutionised not only distributed version-control, it also revolutionised command-line interfaces. Suddenly you could start with remembering a single command called git and find out from here what the full list of subcommands would be. You add --help and it spits out a man page. You have completion out of the box. And everyone can write plugins very easily in just a few lines of shell script. Not only that, but most options can be stored and retrieved in a really straightforward config file, with useful and logical lookup in your project, then home.

It’s really a treasure trove of good ideas, and much more modern than the Java CLI we started copying. So we stopped and rewrote our CLI to copy the Git CLI, and I have to say I’m really glad we did, because we really have a great CLI now.

Discovery

One Command to rule them all, One Command to find them, One Command to bring them all and in the CLI bind them In the Land of Terminal where the Shell lie.

We have a single command called ceylon, and if you do ceylon help or ceylon --help you will get the following:

NAME

        'ceylon' - The top level Ceylon tool is used to execute other Ceylon tools


SYNOPSIS

        ceylon --version
        ceylon [--stacktraces] [--] <command> [<command‑options...>] [<command‑args...>]


DESCRIPTION

        If '--version' is present, print version information and exit. Otherwise '<tool-arguments>' should 
        begin with the name of a ceylon tool. The named tool is loaded and configured with the remaining
        command line arguments and control passes to that tool.

        * 'all' - Compiles ceylon modules for JVM and JS and documents them

        * 'compile' - Compiles Ceylon and Java source code and directly produces module and source archives 
                      in a module repository.

        * 'compile-js' - Compiles Ceylon source code to JavaScript and directly produces module and source 
                         archives in a module repository

        * 'config' - Manages Ceylon configuration files

        * 'doc' - Generates Ceylon API documentation from Ceylon source files

        * 'doc-tool' - Generates documentation about a tool

        * 'help' - Displays help information about other Ceylon tools

        * 'import-jar' - Imports a jar file into a Ceylon module repository

        * 'info' - Prints information about modules in repositories

        * 'new' - Generates a new Ceylon project

        * 'run' - Executes a Ceylon program

        * 'run-js' - Executes a Ceylon program

        * 'src' - Fetches source archives from a repository and extracts their contents into a source directory

        * 'test' - Executes tests

        * 'version' - Shows and updates version numbers in module descriptors

        See 'ceylon help <command>' for more information about a particular subcommand.


OPTIONS

        --stacktraces
            If an error propagates to the top level tool, print its stack trace.


        --version
            Print version information and exit, ignoring all other options and arguments.

Yes it’s a man page, this is what is useful and will help you find your way out of the many ceylon subcommands we have.

A similar thing will happen if you type ceylon compile --help or ceylon help compile.

Completion

We ship with completion support for bash:

$ . /usr/share/ceylon/1.0.0/contrib/scripts/ceylon-completion.bash 
$ ceylon [TAB]
all         compile-js  doc         help        info        run         src         version     
compile     config      doc-tool    import-jar  new         run-js      test        
$ ceylon compile[TAB]
compile     compile-js  
$ ceylon compile --re[TAB]
--rep\=       --resource\=  

As you can see, it’s quite useful, and again, that’s what you expect in this day and age.

Documentation

The tool system that we wrote (well, mostly that Tom Bentley wrote) even allows us to generate HTML, XML and text documentation automatically, for example, the tool documentation pages are entirely generated, and the man pages that we ship in the Ceylon CLI distribution are also generated.

Plugins, plugins

You may not have noticed it but if you type ceylon help on your system, it will likely not include that line:

[...]
DESCRIPTION

        [...]

        * 'all' - Compiles ceylon modules for JVM and JS and documents them

And that’s because, like Git, we support CLI plugins, and ceylon all is a plugin I wrote for myself, that aggregates a bunch of subcommands in a single one.

Ceylon CLI plugins are as straightforward as Git CLI plugins: any executable in your path which starts with ceylon- will be picked up to be a plugin. It will be listed and documented too. If you add documentation in your plugin it will be used.

Here’s my ceylon-all shell script for example:

#!/bin/sh

USAGE='[generic options] module...'
DESCRIPTION='Compiles ceylon modules for JVM and JS and documents them'
LONG_USAGE='ceylon-all allows you to build the specified ceylon modules for the
JVM and JS backends, and generates the API documentation in a single command.'

. $CEYLON_HOME/bin/ceylon-sh-setup

$CEYLON_HOME/bin/ceylon compile $@
$CEYLON_HOME/bin/ceylon compile-js $@
$CEYLON_HOME/bin/ceylon doc $@

As you can see, our CLI will use the USAGE, DESCRIPTION and LONG_USAGE environment variables for completion and documentation. You only have to source the provided $CEYLON_HOME/bin/ceylon-sh-setup shell script to benefit from this. Then you can run whatever you want.

Configuration

There are a lot of options the CLI and IDE have good defaults for, such as the source path, or the output module repository, or the list of module repositories. But when you need something different it gets tiring really fast to have to specify them for every command, so we support configuration files just like Git, in the form of INI files located in your project at .ceylon/config. If that file does not exist or a particular configuration option is not defined in it, we will also try the user’s config in $HOME/.ceylon/config as a fallback.

The syntax is straightforward, take for example the .ceylon/config for the Ceylon SDK:

[repositories]
output=./modules
lookup=./test-deps

[defaults]
encoding=UTF-8

It just specifies that the output repository is modules, the additional lookup repository is test-deps (for test dependencies) and the source encoding is UTF-8.

There’s a ceylon config command which allows you to edit this file if you don’t want to fire up vim for that, and the best part is that this config file is actually used and edited by the IDE too! So no more hassle to keep the CLI and IDE settings in sync.

Welcome to the future! :)