Ceylon 1.2.2 was released back in March, and at the time it shipped with a new feature that did not make a lot of noise, but I think it's worth explaining it a bit more in this blog.

In general Ceylon developers have to download the Ceylon distribution in order to compile Ceylon code, and so do Ceylon users. When you check out a Ceylon project you have to make sure you're using the right version of the Ceylon distribution to compile it, and switching Ceylon distribution can be a bit too involved on some Operating Systems.

To solve that issue, Tako has implemented the ceylon bootstrap tool, very heavily inspired by the Gradle Wrapper, which allows you to generate a small script in your Ceylon project which your users and developpers can use to automatically download the right Ceylon distribution and cache it for future use.

Let's try it out by generating a new Ceylon project (on a machine with a Ceylon 1.2.2 distribution, this is a chicken and egg problem and we have to start somewhere):

$ ceylon new hello-world --ant=no --eclipse=no myproject
Enter module name [com.example.helloworld]: 
Enter module version [1.0.0]:

Now let's check that it compiles and runs:

$ cd myproject
$ ceylon compile,run com.example.helloworld
Note: Created module com.example.helloworld/1.0.0
Hello, World!

Excellent! Now let's add the Ceylon bootstrap tool:

$ ceylon bootstrap
$ ls ceylonb*
ceylonb  ceylonb.bat
$ ls .ceylon/bootstrap/
ceylon-bootstrap.jar  ceylon-bootstrap.properties

As you can see, we've created two ceylonb scripts (one for Unix, one for Windows), and a new configuration folder for those scripts containing a tiny jar (24k) and a settings file which points to the distribution to use:

$ cat .ceylon/bootstrap/ceylon-bootstrap.properties 
#Generated by 'ceylon bootstrap'
#Tue Aug 02 17:29:39 CEST 2016
distribution=https\://downloads.ceylon-lang.org/cli/ceylon-1.2.2.zip

These scripts and folders should be committed to your project code, so they can be used by others.

Now, whenever someone checks your project out, they can just use the provided ceylonb script to work rather than pre-install the Ceylon distribution:

$ ./ceylonb compile,run com.example.helloworld
Downloading Ceylon... 100%
Note: Created module com.example.helloworld/1.0.0
Hello, World!

The first time you run this, it will download the desired Ceylon distribution to ~/.ceylon/dists, but from then on, whenever you invoke ceylonb it will reuse that distribution without downloading anything. Even other projects using the same distribution will be able to use it.

As I said, Gradle Wrapper users are used to this, but still, it's a really good idea and there's no reason why programming languages couldn't do the same!

TL;DR: coming along, won’t be done in time for official GSoC deadline, I’ll just keep working.

(Note: this is a GSoC status report. The entire project is work in progress, and if you’re looking for a something finished, you’ll have to wait until the project is released, at which point there will be a proper announcement.)

For this year’s Google Summer of Code, I am working on adding TypeScript support to the Ceylon JavaScript backend. The goal is to have a tool that, given a TypeScript module (one or more files), produces a Ceylon module for the JavaScript backend. The JS code of that module will just be the TypeScript compiler’s output (plus possibly some metamodel stuff), but the tool will also add model information that will allow you to use the TypeScript module from Ceylon like any other Ceylon module, without needing to use dynamic blocks or having to declare your own dynamic interfaces.

I actually started work on this tool a few months before GSoC, in January this year. I wrote the first iteration of the program in TypeScript in order to be able to interact with the TypeScript compiler (which is also written in TypeScript). The goal of that first iteration is just to be able to load the TypeScript compiler itself, so that I can then use that module (tsc) from Ceylon and write the second (and probably final) iteration of the program in Ceylon.

This first iteration of the program was mostly finished, at least good enough that I could start writing the second iteration, shortly after the GSoC work period officially started. I had hoped to start with the second iteration exactly at the start of GSoC, but while there had been good progress with my own simple test module, tsc held a couple of nasty surprises that I hadn’t anticipated and that I had to work around. It also turned out that I had made a couple of very bad decisions early on:

• I chose to operate on the parsed syntax tree instead of the typechecker model. How I ever expected this to work, I have no idea, but problems quickly became obvious:

  • When I see a type reference, like Element, I have to know if this refers to a type parameter, a type from the same module, or perhaps one from another module. The type I store in the model must be fully qualified.
  • To represent a type with type arguments, the JS model lists the type arguments along with their type parameter name. So to transform a type like Array<string>, I have to know that Array’s first type parameter is called Element.
  • TypeScript features type inference. Of course, I have to know the inferred type so that I can put it into the model.

• To get a first version out quickly, I just wrote directly to the -model.js file. The model is mostly JSON, so this meant some ugly manual comma management when it turned out there are some declarations where I don’t emit anything (such as index signatures).

There are some lovely commit messages in the git log, with phrases like “totally hacky”, “should be enough for now”, “this is where it all falls apart”, “add fake support”, “horrible hack”, and “How could I possibly ever think that it’s possible to write this loader without access to tsc’s model, purely based on the AST?” I am very glad that I get to learn from these mistakes in the second iteration :)

Now, writing the second iteration also turned out to be trickier than I’d thought. I had (still have) a Ceylon version of the TypeScript compiler, the tsc module, but it’s difficult to work with from Ceylon. Two problems mainly held me up:

1. TypeScript supports optional members: { name?: string } is the type of a value that might have a member name (containing a string), but might also have no such member (undefined). The Ceylon runtime doesn’t like this at all and breaks in many different and difficult-to-debug ways; the only solution I found was to not emit such members at all. To access them without “attribute does not exist” typechecker errors, I use eval inside dynamic blocks, which leads us to the second problem.
2. Whenever a dynamic value (such as the return value of an eval call) is assigned to a Ceylon type, the runtime “dresses” the value with the type that it’s being assigned to (adds RTTI). If this is not the most precise type of the value (say, you dressed the value with Node and later want to use it as a VariableDeclaration), you have to jump through some hoops to fix this – a simple assert (is VariableDeclaration node) won’t work, since Ceylon doesn’t like to narrow these dressed types.

I now have workarounds for both problems, but they took a while to find, and of course I shouldn’t have to work around them in the first place, so I’ll have to teach the Ceylon JS backend to “do the right thing” eventually. (I also have to do that in many other cases, since the Ceylon JS “ABI” differs significantly from plain JS – Ceylon classes aren’t instantiated with new, toplevel values compile to functions, arrays aren’t arrays, etc. etc.)

Right now, the second iteration supports:

• toplevel values,
• toplevel functions (no parameters), and
• the string type.

That’s it. But, unlike the first iteration, it supports this without needing to add any JS code (I’ve already taught the compiler to access the toplevel values as values and not functions). Currently, some JS code is added (metamodel stuff), but I’m not sure if I’ll actually keep that. We could say that what, are you crazy, why would TypeScript modules support the metamodel?, and then the JS file would just be the unaltered tsc output, and the tool would only add the -model.js file. This would probably make working with declaration files easier (where the JS isn’t generated by tsc – one of many problems I haven’t even begun to think about yet).

I’m fairly confident that I’ll be able to add support for most “basic” features – type references and other primitive types, type parameters and arguments, parameters, classes, interfaces, methods, attributes – without too much trouble. After that, all bets are off. TypeScript supports some crazy features, like string types ("foo" | "bar" | "baz" is a valid type) and type guards (pet is Fish is a valid return type, a weird inversion of Ceylon’s is Fish pet conditions), and I have no idea how well we can support them. I will also need to make more changes to the JS backend and runtime, and I don’t know how difficult those will be.

There are about four weeks left before the official GSoC deadline, and I’ll be on vacation for one of them. I hope that the “basic” support will be done until then, but there’s zero chance that the project will be in shippable state. I’m not too worried about that – I’ve worked on the project before GSoC started, I’ll continue to work on it after GSoC ends, just like I did with ceylon.ast two years ago. And, just like two years ago, I should have more time after GSoC ends, because GSoC is actually scheduled pretty weirdly for me, where I still have lectures for most of it and it barely overlaps with my semester break. I should be free to work on this project for most of August, September, and the first half of October, and hope to arrive at something that can actually be released somewhere before the end of that period.

TL;DR: This article describes the modularity changes in the Ceylon run-time and distribution, in order to make them lighter at run-time. Skip to the Final runtime dependencies section if you just want the outcome.

Ceylon has featured a modular architecture from the start. Not just for Ceylon users who write modules, but also within the Ceylon distribution. Historically we used to have very few modules, that were directly related to separate Git projects. Adding a new module meant a new repository and lots of changes in the build. Naturally, as the project grew, each of those modules also grew, and got new third-party dependencies, and occasionally adding a feature in one module was made tremendously easier by just adding that "one more" dependency between distribution modules, resulting in a big spaghetti graph of distribution modules that is common in older/evolved systems.

As we initially expected most Ceylon users to run their code using the ceylon run command, we figured that since they have the Ceylon distribution installed, it does not matter if they depend on more modules from that distribution than strictly necessary. Those modules had to be there anyway, so it would not save any bandwidth to reduce those dependencies.

Naturally, we were wrong, and between the Ceylon Eclipse or IntelliJ IDEs, running Ceylon on OpenShift, WildFly, Vert.x or on Android, people started running Ceylon without the distribution installed, using just the standard java runner. It became soon apparent that we had to untangle those dependencies to make the runtime requirements lighter.

Historically we had the following modules:

• common module used by other modules
• typechecker (the shared compiler front-end)
• Java compiler back-end
• JavaScript compiler back-end
• module repository system
• JBoss modules runtime

The model module

When we implemented reified generics, we had to add subtyping to the runtime, so that we'd be able to figure out if is T bar was true or not. The easiest thing at the time was to "just" depend on the typechecker (compiler front-end) which dealt with the language model and subtyping, and the Java compiler back-end, which had infrastructure to load a language model from JVM information such as class files, or in this case reflection.

This essentially made the runtime depend on the compiler front-end and back-end, which we realised was not ideal, so during the Ceylon 1.2 development, we extracted all model description, loading and subtyping to a new ceylon-model module, but we did not have enough time to do more and so these dependencies remained due to other causes.

Supporting Java 9

During our work on supporting Java 9 / Jigsaw modules in Ceylon, it became clear that having kept our "fork" of javalang tools (that we use for javac) under its original package name would not work anymore, we renamed its package and used the opportunity to prune away parts of the java tools we did not use. We also extracted the class-file reader part to its own module so we could use it outside of the compiler to remove our dependency to jandex (a class-file scanner).

Finally, when we created the ceylon jigsaw tool (which populates a folder with the jar files required by a Ceylon module, to run it on a Java 9 VM) it became evident that the runtime still depended not just on the compiler front-end and Java back-end, but even on the JavaScript back-end, which frankly made little sense in most JVM executions.

These dependencies were due to the Ceylon Tool Provider API having snuck into the ceylon.language module as a convenience (at the time). Since that allowed you to compile and run Ceylon programmatically for both Java and JavaScript back-ends, it had to depend on the tools.

We decided to split the Ceylon Tool Provider into its own module and got rid of the final dependencies from the language module to the compilers and typechecker, but had no more time to get rid of further dependencies such as JBoss Modules and Aether in time for Ceylon 1.2.2.

Supporting Android

Initial work on running Ceylon on Android revealed that what passes for small dependencies on ordinary JVM executions, or even on Java EE deployments, was not an option on Android where every method counts.

At this point we had to bite the bullet and make every non-required transitive dependency go.

We noticed that the old common module had grew to include the Command-Line Tooling API that makes the ceylon command and its subcommands and plugins work. That in turn depended on a Markdown renderer used by ceylon doc. It was pretty trivial to extract it to its own module because this was never used in Ceylon user programs.

Next in line was our Shrinkwrap Resolver dependency, which our module repository system uses to interoperate with Maven repositories. This was a fat-jar with all its dependencies included, including some Apache Commons modules, and an outdated version of Eclipse Aether. That fat-jar had already been problematic in our Maven module, which already had its version of Aether, so getting rid of the fat-jar was a good idea. We also realised that some of its Apache Commons dependencies were already included outside the fat-jar in our distribution repository, so there was that duplication to fix too.

So what we did was remove the Shrinkwrap Resolver dependency and use Aether directly, by incorporating all its subcomponents into our distribution. It turns out that because the latest version of Aether requires Google Guava, our distribution grew in size rather than shrink (that jar is huge). But to offset that, we made the Aether dependency optional, and made sure it was possible to run Ceylon without it as long as there was some compilation step beforehand that provided all the Maven dependencies that you may use in interop. ceylon fat-jar or ceylon jigsaw would do that for you, for example.

Our module repository system also provided support for writing to WebDAV or Herd repositories, which required some dependencies on Apache Http Client or Sardine, and we made these dependencies optional as well, because at runtime your Ceylon program is very unlikely to write to HTTP repositories. This is something only the compiler and other tools do.

We also removed a dependency to JBoss Modules from the language module using abstraction, since that platform was optional and never used on Android or other flat-classpath runtimes.

Finally, the language module only had one dependency left on the (much slimmer) module repository system via the presence of the Main API in there, and we moved that class to its own module.

Final runtime dependencies

After all this pruning, the language module on the JVM is back down to requiring the following set of transitive dependencies:

• common (small and free of tooling and dependencies)
• model (which depends only on the class-file reader)
• class-file reader

So your Ceylon module will only depend on four jars (these three and the language module), the sum size of which is 2.4 Mb, which is much smaller than initially, and has dramatically less methods, at around 17148 methods. This is still too much, but can be brought down by tooling such as ProGuard to remove unused classes. Remember this includes a runtime for an entire language, so it's not that big, all things considered.

SDK changes

In order to be able to use Ceylon's HTTP client on Android, we also split up the ceylon.net module from the Ceylon SDK into client and server modules. Otherwise the HTTP server and its dependencies were too much drag for Android's method count.

Ceylon 1.2.3 is looming closer, and so we should start talking about some of the new features that it will contain. One of those new features is the new ceylon fat-jar command.

ceylon fat-jar my.module/1 lets you generate a jar which contains the my.module/1 module and every required dependency. Once you have that jar, you can execute it with java -jar my.module-1.jar and it will execute the module.

Naturally, you can customise the function or class to execute with the --run option, but the default is the run method in your module (my.module::run in our case).

The modules are executed in a "flat classpath" mode, without JBoss Modules or any sort of module isolation, and with the metamodel already set up statically when the fat jar was created, so it's a little bit different to the older alternative of running Ceylon modules using the Main API and ceylon classpath tool, which did not deal with packing all the dependencies. It is also different to the ceylon run --flat-classpath in that it does not require a module repository at run-time, since ceylon fat-jar packs it all together.

Try it out, it's pretty useful to distribute Ceylon programs and run them in places without having to install the Ceylon distribution.

In my last post, I explained how you can use Ceylon in Apache Cordova to write applications for every mobile platform, including iOS and Android. This time, with many apologies for writing it late (“next week” turned into next month), I will explain how to use the Ceylon IntelliJ plugin to write native Android applications in Ceylon in Android Studio.

Getting started with Ceylon on Android Studio

To start writing Ceylon applications in Android Studio, follow these steps:

• Download Android Studio
• Start it
• Create a new application by clicking on Start a new Android Studio Project
• You can use these values for Application name: CeylonDemo
• And for Company domain: android.example.com
• Next, select an Empty activity, with Activity Name: MainActivity
• Click Finish and wait for the project to be created

At this point you have an Android project open, but we still haven't had time to install the Ceylon plugin, so let's do this right now:

• Click on File > Settings > Plugins > Browse Repositories
• Then on Manage Repositories > +
• Add this repository: https://downloads.ceylon-lang.org/ide/intellij/development/updatePlugins.xml
• Now click on Install Ceylon IDE

You will likely need to restart Android Studio, so do that.

Next we're going to convert our Android project to a Ceylon Android project:

• In the Android view, Right-click on app > Configure Ceylon in this Module
• Click OK on the resulting configuration dialog

This will set up the Ceylon plugin, and will add most of what you need in your Gradle build to build Ceylon Android applications. You now have your Ceylon sources in app/src/main/ceylon and it includes a module descriptor and an empty activity:

At the moment, this requires a Ceylon 1.2.3 distribution to build, and since it's not released yet you're going to have to either build one yourself (just the Getting the source part), or download a nightly build. Once you have it, edit app/build.gradle near then end to add ceylon > ceylonLocation and make it point to where you installed your distribution (it needs to point to the Ceylon binary, not just the distribution root):

ceylon {
    // ...
    ceylonLocation ".../ceylon/dist/dist/bin/ceylon"
}

Make sure you click on Sync now to sync your Gradle build.

Now, there's a bug we're in the process of fixing which fails to detect the exact version of the Android SDK tooling and modules, and so depending on which version of the Android Tools you're using you may have to sync the imports of com.android.support:appcompat-v7:23.1.1 in app/build.gradle (in dependencies) and in the Ceylon module descriptor in module.ceylon. Make sure the Ceylon import version is the same as the Gradle import version, because the Gradle build is what makes it available to Ceylon, due to Android's peculiarities.

In order to finish the conversion, make sure you delete the Java activity (since we're going to keep the Ceylon one), in Project Files, delete app/src/main/java.

Due to another pending plugin fix, you may have to click on Tools > Ceylon > Reset Ceylon Model at this point so that the Ceylon plugin gets synchronised with all these past changes (don't worry we're fixing this at the moment).

The good news is we're already able to click on Run app and try this in the emulator, but we're going to make it a little more interesting.

Customising your Ceylon Android activity

We're going to be displaying a list of Ceylon modules published on Ceylon Herd, so we will make use of the Ceylon SDK, and in particular you will have to edit module.ceylon to add the following imports:

import ceylon.http.client "1.2.3";
import ceylon.uri "1.2.3";
import ceylon.json "1.2.3";
import ceylon.collection "1.2.3";
import ceylon.interop.java "1.2.3";

Next, we're going to turn our MainActivity into a ListActivity and run an asynchronous task to connect to the Herd REST endpoint, so edit MainActivity.ceylon with this:

import android.os { Bundle, AsyncTask }
import android.app { ListActivity }
import android.widget { ArrayAdapter, ListAdapter }
import android.support.v7.app { AppCompatActivity }
import ceylon.interop.java { createJavaStringArray }
import java.lang { JString = String }
import android { AndroidR = R }
import ceylon.language.meta { modules }
import ceylon.uri { parseUri = parse }
import ceylon.http.client { httpGet = get }
import ceylon.json { parseJson = parse, JsonObject = Object, JsonArray = Array }
import ceylon.collection { MutableList, LinkedList }

shared class MainActivity() extends ListActivity() {

    class LoadModules() extends AsyncTask<String, Nothing, List<String>>() {
        shared actual List<String> doInBackground(String?* uris){
            assert(exists uri = uris.first);
            value response = httpGet(parseUri(uri)).execute();
            value modules = LinkedList<String>();
            assert(is JsonObject json = parseJson(response.contents),
                    is JsonArray results = json["results"]);
            // Iterate modules
            for(res in results) {
                assert (is JsonObject res); // Get the list of versions
                assert (is String name = res["module"],
                        is JsonArray versions = res["versions"]);
                modules.add(name);
                print(name);
            }
            return modules;
        }
        shared actual void onPostExecute(List<String> result){
            print("Got result: ``result``");

            ListAdapter adapter = ArrayAdapter<JString>(outer, AndroidR.Layout.simple_list_item_1,
                createJavaStringArray(result));
            listAdapter = adapter;
        }
    }

    shared actual void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.Layout.activity_main);

        LoadModules().execute("https://modules.ceylon-lang.org/api/1/complete-modules?module=ceylon.");
    }
}

Now edit app/src/main/res/layout/activity_main.xml to change the activity type to a list activity:

<ListView
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:id="@android:id/list"/>

And lastly request the network permission for your app, since we're hitting a web service, by adding this to app/src/main/res/AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET"/>

That's all you need, now just click on Run > Run app and watch your Ceylon application display the list of modules in the emulator:

Some technical info

The Ceylon IntelliJ plugin has not been released yet, but a preview is forthcoming really soon. You will see it's already quite advanced when you try this out. Don't hesitate to report any bugs, or better yet, contribute fixes :)

Most of it works well enough for Android, except the caveats noted above, and the fact that Android Studio does not yet recognize Ceylon classes, so they will be marked as errors in the .xml files that refer to them, and when you run your application it will report an error:

Could not identify launch activity: Default Activity not found. Error while Launching activity

It only means it could not start your application, you will have to click on it to start it in the emulator. But the deployment worked. We're fixing this at the moment, so it will only improve.

If you want to revert to the Ceylon Eclipse IDE to edit your Ceylon Android application, you can, it will work once you have your project set up with Android Studio. It's much easier to use it to set it up so all the Gradle config is just right. Once that is done, you can use Eclipse if you want, and use $ ./gradlew assembleDebug to build your APK.

This work depends on changes we've made in Ceylon 1.2.3 (to be released soon) which adds support for jars which provide alternate smaller JDKs (such as the Android jar), improvements in modularity so that the created applications depend on much fewer runtime Ceylon jars than before, fixes in the runtime metamodel to support Android runtimes, and several other tweaks. I will probably write an account of all that in a future blog entry.

This work also depends on the Ceylon Gradle plugin written by Renato Athaydes, and on a new Ceylon Gradle Android plugin which adds support for Ceylon in Android applications. This plugin is by no means finished, and in particular does not yet support incremental compilation (even though the Ceylon IDE and compiler do). It also does not yet support the latest Android Instant Run feature. Again, please report issues or better, contribute pull-requests :)