Posts By Joram Barrez

Running Flowable on CockroachDB

What is CockroachDB?


CockroachDB is a project I’ve been keeping an eye on for a while. It’s a an open-source, Apache 2 licensed, database system(Github link). At it’s core its a key-value store that scales horizontally. But what makes it really interesting, is that 1) it supports SQL by using the Postgres wire protocol and 2) has full ACID semantics and distributed transactions. If you’re interested in how they achieve this, make sure to read the technical posts at the CockroachLabs blog (I admit, sometimes it’s not for the faint-of-heart ;-)). Do note that it is still a distributed system and thus follows the CAP theorem, more specifically it is a CP system.

It’s still early days, as you’ll read in their FAQ, as many things are not optimized yet. However, now that they recently added basic support for joins, I figured I should give it a spin with the Flowable engine. In this post I’ll show how easy it is to run the Flowable v6 process engine on CockroachDB.

(Sidenote: I love the name! For people that don’t understand it: cockroaches are one of the few creatures on earth that can survive something like a nuclear blast. Quite a resilient little animal … something you’d like for your data too 🙂 )


The getting started docs on the CockroachDb website are quite clear, but for clarity here are the steps I followed:

  • Download latest CockroachDB tarball (or whatever your system needs)
  • Untar and start first node:
    •  ./cockroachdb start
  • Start a second node:
    • ./cockroach start –store=node2 –port=26258 –http-port=8081 –join=localhost:26257
  • Start a third node:
    • ./cockroach start –store=node3 –port=26259 –http-port=8082 –join=localhost:26257

Hurray, you’ve now got a cluster of three nodes running, which will happily replicate data among each other. There is a nice admin app which is running on 8080 and gives an overview of the cluster:


Next step: we need a database for the Flowable engine. Creating the database and granting permissions to the default user (maxroach) is done via the CockroachDB SQL shell:

./cockroachdb sql
> GRANT ALL ON DATABASE flowable TO maxroach;

Sadly, CockroachDB hasn’t implemented the JDBC metadata feature yet, which the Flowable engine uses for automatically creating the database schema. Also, I couldn’t quite get foreign keys to work properly in a few cases, so I copy/pasted the Flowable SQL scripts and removed those. The file is uploaded on Github.

Also, this means that currently you need to create the database schema “manually”. If you’re using a bash terminal, you can download the script above from github and feed it to the CockroachDB SQL shell as follows. Alternatively, you can paste it into the SQL shell.

sql=$(wget -q -O -)
./cockroach sql –database=flowable –user=maxroach -e “$sql”

Flowable on CockroachDB

The database is now ready. Time to boot up a Flowable engine using this database as data store. All source code is available on Github:

As CockroachDB uses the Postgres wire protocol, we simply need to add the Postgres JDBC driver to the pom.xml:


I’m using the current v6 master branch here, which isn’t released yet. You can build it yourself easily though by cloning the flowable-engine project and doing a ‘mvn clean install -DskipTests’ in the root. The configuration file used for the engine is quite simple, and it looks exactly like connecting to a regular Postgres relational database. Do note I’m “cheating” a bit with the databaseSchemaUpdate settings to avoid the auto-schema check.

<property name="jdbcUrl" value="jdbc:postgresql://" />
<property name="jdbcDriver" value="org.postgresql.Driver" />
<property name="jdbcUsername" value="maxroach" />
<property name="jdbcPassword" value="" />

<property name="databaseSchemaUpdate" value="cockroachDb" />

The process definition we’ll use is a simple demo process that exercises a few things like user tasks, service tasks, subprocesses, timers, etc:


The following snippet shows how the Flowable API is used in a few different ways. If you follow on the CockroachDB admin UI, you’ll see that traffic goes up for a while. What happens here is:

  • Lines 3-9: booting up the Flowable process engine using the config file from above and getting all the services
  • Line 11: Deploying the process definition
  • Lines 15-19: Starting 100 process instances
  • Lines 24-33: Finishing all tasks in the system
  • Line 35: Doing a historical query

So, as you can see, nothing fancy, simply touching various API’s and validating it all works on CockroachDB.

public static void main(String[] args) {

  ProcessEngine processEngine = ProcessEngineConfiguration

  RepositoryService repositoryService = processEngine.getRepositoryService();
  RuntimeService runtimeService = processEngine.getRuntimeService();
  TaskService taskService = processEngine.getTaskService();
  HistoryService historyService = processEngine.getHistoryService();

  System.out.println("Process definitions deployed = " + repositoryService.createProcessDefinitionQuery().count());

  Random random = new Random();
  for (int i=0; i<100; i++) {
  Map<String, Object> vars = new HashMap<>();
    vars.put("var", random.nextInt(100));
    runtimeService.startProcessInstanceByKey("myProcess", vars);

  System.out.println("Process instances running = " + runtimeService.createProcessInstanceQuery().count());
  LinkedList<Task> tasks = new LinkedList<>(taskService.createTaskQuery().list());

  while (!tasks.isEmpty()) {
    Task task = taskService.createTaskQuery().taskId(tasks.pop().getId()).singleResult();
    if (task != null) {

    if (tasks.isEmpty()) {

  System.out.println("Finished all tasks. Finished process instances = "
    + historyService.createHistoricProcessInstanceQuery().finished().count());


The output is exactly as you’d expect (and exactly the same as running it on a relational database).

Process definitions deployed = 1
Process instances running = 100
Completed 10 tasks
Completed 20 tasks

Completed 400 tasks
Finished all tasks. Finished process instances = 100


It’s almost trivial to run the Flowable process engine on CockroachDB, most specifically by the excellent SQL layer and relational support the developers of CockroachDB have added. There’s still a way to go (as you’ll read on their blog), but it’s certainly a cool piece of technology already right now! And who doesn’t like horizontal scalability without sacrificing ACID transactions? It’s a perfect fit for the use cases of a process engine.

I’m going to continue to keep a close eye on the CockroachDB project, as the combination with Flowable shows a lot of potential. And, as you know me, I’m also really looking forward, once they start focusing on performance, to run some benchmarks :-).

Flowable REST API now includes Swagger docs

More and more libraries and frameworks are adding Swagger support nowadays. For those who don’t know Swagger: it is a standard specification that is part of the OpenAPI initiative. The core of Swagger is a so-called specfile which is a json (or yaml) document that describes and documents for example a REST API. By using Swagger annotations on the Spring MVC REST controllers, the docs and logic stay close together so it’s easy to maintain. A Maven plugin then processes these annotations and generates this specfile.


The benefits of Swagger are quite clear:

  • Documentation, obviously
  • Auto-generation of client SDK’s. For example, suppose you are using PHP (or any other language/environment) and want to work with the Flowable API’s, then you can generate a client library using the specfile.
  • API discovery or inclusion in an API directory

Of course, such a json specfile is kind of boring to read, so the Swagger UI has been added to the Flowable REST application. This is an html/javascript application that parses and displays the REST API docs in a user-friendly way. The Swagger UI will be accessible under /docs when running the Flowable REST application. You can also the same UI to actually interact with the API (which imho sure is easier than having to look up how curl works for the xth time ;-))

Announcing Flowable


For the many of you that are following me on Twitter it’s probably not news anymore, but I wanted to make sure that all readers of my blog are in the loop (and of course, thanks for visiting my humble spot in the interwebs!).

Last Thursday, Flowable was announced, a fork of Activiti. More details about it are on the Flowable website:

The Road Goes Ever On

As part of the announcement, version 5.22 was released.

The next point on the list of priorities is releasing a new version of the v6 engine and UI’s, this hopefully should be the last version before the final one. The engine has been stable for a few months now, but there are a few areas that need to be finished – mostly in modules outside the engine, and also in the UI.

How to use transient variables in Activiti

A feature that has been requested quite a bit – transient variables – has landed in the Beta3 of Activiti v6 released yesterday. In this post, I’ll show you an example on how transient variables can be used to cover some advanced use cases that weren’t possible (or optimal) before.

So far, all variables in Activiti were persistent. This means the variable and value are stored in the data store and historical audit data is kept. Transient variables on the other hand act and behave like a regular variable, but they are not persisted. Beyond not being persisted, the following is special to transient variables:

  • a transient variable only survives until the next ‘wait state’, when the state of the process instance is persisted to the database.
  • a transient variable shadows a persistent variable with the same name.

More detailed information about transient variables and the API’s can be found in the documentation.


The process definition that we’ll use to demo some bits of the transient variables is shown below. It’s a fairly simple process: the idea is that we’ll ask some things like keyword and language from the user and use it to do a GitHub API call. If successful, the results are shown to the user. It’s easy to write a UI for this (or use the new forms in the Beta3 angularJS app), but in this post we’ll focus on the code only.

The BPMN 2.0 xml and code can be found on this Github repo:

Screenshot from 2016-09-01 11:44:50

Let’s walk through the process together. The process starts by providing some input from the user about what should be searched on (usually this would be done using a start form).


Map<String, Object> variables = new HashMap<String, Object>();
variables.put("keyWord", "workflow");
variables.put("language", "java");
ProcessInstance processInstance = runtimeService.startProcessInstanceByKey("githubsearch", variables);

The variables we pass when starting the process instance are regular variables. They are persisted and audit history will be kept, as there is no reason why this shouldn’t be the case.

The first step that is executed is the ‘execute HTTP call’ step, which is a Service Task with a Java delegate:

<serviceTask name="Execute HTTP call" activiti:class="org.activiti.ExecuteHttpCallDelegate"></serviceTask>

Java code:

public class ExecuteHttpCallDelegate implements JavaDelegate {

    public void execute(DelegateExecution execution) {

        String keyword = (String) execution.getVariable("keyWord");
        String language = (String) execution.getVariable("language");

        String url = "";
        url = String.format(url, keyword, language);
        HttpGet httpget = new HttpGet(url);

        CloseableHttpClient httpclient = HttpClients.createDefault();
        try {
            CloseableHttpResponse response = httpclient.execute(httpget);

            execution.setTransientVariable("response", IOUtils.toString(response.getEntity().getContent(), "UTF-8"));
            execution.setTransientVariable("responseStatus", response.getStatusLine().getStatusCode());


        } catch (Exception e) {



Here, we’re doing a simple HTTP get against the GitHub API, using the ‘keyword’ and ‘language’ variables we’ve passed on process instance start. Special here is on line 16 and 17 that we’re storing the response and response status in transient variables (that’s the setTransientVariable() call). The reasons for choosing transient variables here are

  • The json response from the Github API is very large. It can be stored in a persistent way of course, but this won’t be good for performance.
  • From an audit point of view, the whole response matters very little. We’ll extract the important bits later from that response, and those will be stored in the historical data.

After getting the response and storing it in a transient variable, we pass the exclusive gateway. The sequenceflow looks like this:

<sequenceFlow ... >
    <activiti:executionListener event="take" class="org.activiti.ProcessResponseExecutionListener"></activiti:executionListener>
  <conditionExpression xsi:type="tFormalExpression"><![CDATA[${responseStatus == 200}]]></conditionExpression>

Note that for the sequence flow condition there is no difference when it comes to using a transient or non-transient variable. A regular getVariable will also return the transient variable with the name, if set (this is the shadowing part in the docs mentioned above). A getTransientVariable also exists, when only the transient set of variables should be consulted. Anyway: for the condition: no difference at all.

You can also see that the sequence flow has a (hidden in the diagram) execution listener. The execution listener will parse the json response, select the relevant bits and store these in a transient array list. This is important, as you’ll read below the code.

public class ProcessResponseExecutionListener implements ExecutionListener {

    private ObjectMapper objectMapper = new ObjectMapper();

    public void notify(DelegateExecution execution) {

        List<String> searchResults = new ArrayList<String>();

        String response = (String) execution.getVariable("response");
        try {
            JsonNode jsonNode = objectMapper.readTree(response);
            JsonNode itemsNode = jsonNode.get("items");
            if (itemsNode != null && itemsNode.isArray()) {
                for (JsonNode itemNode : (ArrayNode) itemsNode) {
                    String url = itemNode.get("html_url").asText();
        } catch (IOException e) {

        execution.setTransientVariable("searchResults", searchResults);


The reason for storing the list as a transient variable is an important one. As you can see in the diagram, a multi instance subprocess follows. A subprocess often takes a collection variable to create the instances. So far, this was a persistent variable, in the form of a java-serialized ArrayList. I’ve never liked that (I’ve always used delegateExpressions with a bean if I had to do it before). This has always bothered me a bit. Now, the arraylist is transient and won’t be stored in the data store:

  <subProcess name="subProcess">
    <multiInstanceLoopCharacteristics isSequential="false" 
          activiti:collection="searchResults" activiti:elementVariable="searchResult" />   

Do note that the ‘searchResult’ variable above will be a persistent variable.

Note that the transient variables will be there until the user task is reached and the state is stored in the data store. It’s also possible to pass transient variables when starting a process instance (which could have been an option here, but I think storing user input is a thing you’d want in your audit data).

If you’d run the process instance like this for example:

Map<String, Object> variables = new HashMap<String, Object>();
variables.put("keyWord", "workflow");
variables.put("language", "java");
ProcessInstance processInstance = runtimeService.startProcessInstanceByKey("githubsearch", variables);

List<Task> tasks = taskService.createTaskQuery().processInstanceId(processInstance.getId()).list();
for (Task task : tasks) {
  System.out.println("Current task : " + task.getName());

Which gives as example output (limited to 5 results):

Current task : Review result
Current task : Review result
Current task : Review result
Current task : Review result
Current task : Review result

The user could now look into the details of each of the results and continue the process.

Last Words

As you can imagine, there are quite a few use cases for transient variables. I know that for many people this was an important feature, so I’m glad it’s out there now. Feedback and comments of course, as usual, always welcome!

Benchmarking the message queue based Activiti Async Executor

A bit of history

One thing that never ceases to amaze me is how Activiti is being used in some very large organisations at some very impressive scales. In the past, this has led to various optimizations and refactorings, amongst which was the async executor – replacement for the old job executor. For the uninitiated: these executors handle timers and async continuations in process instances. The introduction of the async executor boosted performance significantly. However, last year at the community event in Paris we learned that when dealing with an enormous amount of jobs, the queries used by the executor could lead to the need for table scans. Which is never a good thing.

So, there was one thing to do before finalising version 6, and that is refactoring the async executor such that all queries it used were dead simple. This did mean splitting the job data into various tables that match the different types and states, while still keeping API compatible with previous releases.

In the last couple of months, that’s been implemented (amongst many other things), with some nice results and some new nice APIs that enrich the platform. If you are interested in how it all works, go and check the online docs or check the source code on the v6 branch.

The architectural design is of course influenced by what is learned from the past two implementations, but it’s also heavy influenced by concepts from message queueing systems. One of the design goals was that it should be super-easy to plug in a message queue and run with it, as there was a gut feeling that this would be beneficial for performance.

Making the async executor work together with a message queue proved to be almost trivial due to the new architecture. If you are interested in the implementation, check the section in the docs on this very topic.

And, of course, you know me, I just wanted to benchmark these two executor implementations against each other 🙂

Benchmark project

You can find the code I used on Github:

Basically, what it does is run the with a configuration properties file.

  • Booting up a process engine with a decent config (I’ve seen some Activiti benchmarks recently online that benchmarked Activiti’s performance without using a decent connection pooled datasource. Sigh, but anyway.)
  • If running as ‘producer’, 10 000 process instances will be started, one every 10 milliseconds. Periodically stats will be printed to the console.
  • If running as ‘executor’, the process engine is configured to have the async executor enabled.
  • There can be an arbitrary number of producers/executors, but all go to the same database.

The process definition used in the project looks as follows:

Screenshot 2016-07-20 00.37.54

Important to note (and not visible on the diagram) is that all service tasks are asynchronous in this non-trivial process definition. The service tasks after a parallel fork are configured to be exclusive, as are the joining parallel gateways. There are two timers here, where the one on the user task is 1 second and the one on the subprocess is 50 minutes. All in all, when starting a process instance, it leads to 27 jobs needing to be executed to reach the end. For 10 000 instances, this means we’re testing effectively the throughput of 270 000 jobs.

Note that, as with any benchmarks, raw numbers say something but not everything. It all depends on the server hardware, the actual process definitions and many other small bits. Relative numbers however, they do teach us a lot, if the exact same code is being executed on the exact same hardware. Keep that in mind when reading the next sections.

Test environment

All benchmarks were run on Amazon Web Services (AWS), using EC2 servers for the producers/executors  and using RDS PostgresQL (as Postgres is an awesome database and very easy to set up) for the database on a r3.4xlarge (16 vCPUs, 122 GiB memory) .

Following EC2 configs were used

  • RDS (postgres) : r3.4xlarge (16 vCPUs, 122 GiB memory)
  • Producer engine: c3.4xlarge (16 vCPUs, 30 GiB memory)
  • Executor engine: c3.8xlarge (32 vCPUs, 60 GiB memory)

All severs ran in the EU-West zone. All test results thus have real network latencies (None of that running on localhost benchmarking and thus skipping networking as often seen online). When running the project above, 8GB was given to the JVM.

The metric we’ll use is throughput of jobs, expressed in jobs/second. Simply said, after test run, we verify the data in the database to be correct (i.e. 10K finished process instances), take the first start time and the last end time which gives us x seconds. The throughput is then x/270000 (as we know each process instance equals to 27 jobs).

Baseline Measurements

The first thing benchmarked was the ‘baseline’, meaning the regular async executor that is backed by a threadpool (i.e. the improved design of the async executor in v5). For this test we used 2 servers, with following configurations (note: 6.0.0.Beta3 here is actually the snapshot version):

Activiti version 6.0.0.Beta3 6.0.0.Beta3 6.0.0.Beta3 5.21.0
Producer engines 1 1 1 1
Executor engines 1 1 2 2
# threads in pool 32 10 10 10
Blockingqueue size 256 100 100 100

image (4)

Some interesting observations:

I assumed config A would be better then config B, as the machine had 32 CPU’s after all, so matching the threadpool number of threads with this would make sense. However, config B, which has a very similar setup except only 10 threads and a smaller blockingqueue beats it significantly (310 vs 210 jobs/second). A possible explanation could be that 32 threads is too much contention? I do remember when choosing the default of ’10’ back in the day, we did some benchmarks and 10 was the ‘magic number’ where throughput was best (but I do think it will depend on the machine used.

I expected that adding another executor node would have more impact, after all we’re adding a 32 CPU machine into the mix, but the gain is minimal (310 to 326). We’ll learn why and fix this in a later stage in this article.

Config D, using Activiti version 5.21.0 uses the same setup as config C. However, the improved async executor of version 6 clearly wins here (326 vs 266). Which was of course what we hoped for :-).

So far, our best result is 326 jobs/second (and using two servers).

Variations on the baseline

Given the setups above, one can ask what the influence is when running a mixed producer/executor. Which is the default Activiti engine way of running it: the engine will both be responsible for starting process instances and executing them now. This is config E (the same as config C, except both engines are now producers/executors) and the result is shown below. And it’s clearly less performant. One explanation could be that the machine is already using 10 threads to start process instance every 10 ms, which probably leads to quite a bit contention with the 10 threads of the async executor. Probably this setup can be tweaked a lot to get better numbers, but that wasn’t the goal of this blog. But the result is interesting nonetheless.

image (3)

So given that two executor engines were better than one, the logical thing is to try three executors. This is config F.

Similar to going from one to two executors, the throughput goes up. But not in a spectacular linear way.

image (2)

Introducing the Message Queue based Async Executor

Time to switch to the message queue based async executor, now we have our baseline numbers. I chose the latest version of ActiveMQ, as I’m familiar with it and setting it up is super-easy. I did not spent any time tweaking ActiveMQ, switching persistence strategies or trying alternatives. So there’s probably some margins to gain there too.

In the benchmark project, I used Spring with the following config: The reason for going with Spring is that the MessageListenerContainer gives an easy way to have a message queue listener work nicely with multiple threads (which application servers like JBoss would give you otherwise). More specifically, the concurrenConsumers setting of the MessageListenerContainer allows setting the number of threads being used for listening to messages in a smart way. Yes, this class does have a lot of properties that probably can influence the results for the better, but again that was not the point here. Relative numbers, remember.

We’re using a similar setup as config C (our best result so far with two servers), for this config, called config G: 1 producer engine, 2 executor engine. Note that we’re also adding a ‘queue server’ to the mix now, that’s using a c3.8xlarge machine (32 vCPUs, 60 GiB RAM) like the executor engine server.

The results are below … and they are simply awesome: the message queue async executor in a equivalent setup (but with an extra message queue server) is four times faster than the threadpool based async executor.

image (5)

One small implementation note: we had to switch to the UUID ID generator, as the throughput was too high for the default one. Bearing in mind that the UUID generator is slower than the default, the results are even more awesome (as we’re really talking about milliseconds here).

Interesting observations!

If you’d run the benchmark project, you’d see that it periodically spits out some stats so you can follow how many jobs, timers, user tasks, historic activity instances, process instances, etc. are in the system.

While running the message queue setup, one pattern became very clear from these numbers. The threadpool based asyncexecutor was finishing process instances quicker (i.e. after like 1 minute, we saw a batch of process instance being completed), while for the message based async executor, the process instances were practically all finished in one big burst at the end. This indicates that the latter would spread the execution of process instance activities more, while the thread-based would hammer on until one is finished.

Some discussions in the team led to the explanation for this: the threadpool based one will always pass the next async job to the executor, while the message based one puts it on the queue, where already thousands of messages are waiting. Add now the fact that we have quite a bit exclusive async jobs for the process instance, this means that for the threadpool based one, many threads are trying to get the process instance lock, but failing as an exclusive one is being executed. However, the job was unacquired and quickly picked up again. For the message queue based one, they are added again to the end of the message queue. Which has thousand of other messages waiting. When it comes back to executing this particular message, the exclusive lock is most likely already long passed.

This led to some refactoring in the threadpool based async executor: instead of simply releasing the lock on the job, the job is deleted and reinserted, effectively mimicking the queue behavior. This is the fix:

Benchmarking these in an exact same setup as config C, called config H (1 producer, 2 executors), shows us that this simple fix gives a 34% boost to throughput! We now have a new baseline 🙂

image (6)

Even better message queue async executor results

So, in the message queue result (config G), we used a fairly conservative setting of 10 threads for listening to messages. The idea was that we also had 10 threads for the threadpool. Of course, a message queue consumer is fundamentally different from threads that poll: such a consumer has a persistent connection with the queue and the queue broker actually pushes work to its consumers. This should be more efficient. So we’ve tried following configurations, where we vary the amount of consumers (and thus threads used to consume) and executor nodes.

Producer engines 1 1 1 1
Executor engines 2 2 3 3
# consumers / engine 32 64 32 64

image (9)

So one nice observation is that adding more consumers is super effective. We’re getting up to a throughput of 2222.9 jobs/second. That is blazingly fast if you ask me, and five times as fast as the threadpool based async executor. 

Sadly, adding more executor machines to the mix is actually bad for performance. I think that the bottleneck now becomes the database and how it handles all this concurrency going on at high scale. Of course, I did not tweak the database at all, just a regular RDS postgres instance. Or experiment with Aurora or Oracle (which got the best results in my previous benchmarks). However, the point here was relative numbers, not squeezing out the last bit of throughput. I think the relative number point has been made 🙂


The numbers speak for themselves: the new message queue based async executor beats the threadpool based async executor hands down. Does this mean you have to switch immediately? No, the regular async executor is also seriously fast (436 jobs/second is still fast), but more importantly, the setup is way simpler, as the Activiti engine takes care of everything. Adding a message queue to your project means additional complexity: another thing that can fail or crash, extra monitoring, maintenance, etc. However, when you’re doing a lot (and I do mean _a lot_) of async jobs, and you’re hitting the limits of what the default async executor can do, it’s nice to know there’s an alternative.

Let’s also not forget the other conclusion made here: the new async executor implementation in version 6 is a major improvement over the version 5 one!

Further Work

The current implementation is Spring/JMS only. However, the implementation is trivial to port to other systems and/or protocols (application servers, STOMP, AMPQ, AWS SQS, etc.). Feedback is appreciated as to what would be a popular next choice 🙂

Interestingly, this message queue based async executor makes implementing ‘priority queues’ very simple. Priority queues are a feature that many of our large users have asked for: to give certain process definitions/instances/on certain conditions/… priority vs regular jobs. It’s easy to imagine how to set up multiple queues and/or allocate less or more consumers to give certain use cases priority.

How the Secure Scripting in Activiti works

One of the prominent features of the recent Activiti 5.21.0 release is ‘secure scripting’. The way to enable and use this feature is documented in detail in the Activiti user guide. In this post, I’ll show the details of the implementation and what it’s doing under the hood.

The Problem

The Activiti engine has supported scripting for script tasks (and task/execution listeners) since a long time. The scripts that are used are defined in the process definition and they can be executed directly after deploying the process definition. Which is something many people like. This is a big difference with Java delegate classes or delegate expressions, as they generally require putting the actual logic on the classpath. Which, in itself already introduces some sort of ‘protection’ as a power user generally only can do this.

However, with scripts, no such ‘extra step’ is needed. If you give the power of script tasks to end users (and we know from some of our users some companies do have this use case), all bets are pretty much off. You can shut down the JVM or do malicious things by executing a process instance.

A second problem is that it’s quite easy to write a script that does an infinite loop and never ends. A third problem is that a script can easily use a lot of memory when executed and hog a lot of system resources.

Let’s look at the first problem for starters. First off all, let’s add the latest and greatest Activiti engine dependency and the H2 in memory database library:


The process we’ll use here is trivially simple: just a start event, script task and end. The process is not really the point here, the script execution is. Screenshot from 2016-06-13 20:16:21

The first script we’ll try does two things: it will get and display my machine’s current network configuration (but there are obviously more dangerous applications of this idea) and then shutdown the whole JVM. Of course, in a proper setup, some of this will be mitigated by making sure that the user running the logic does not have any rights that matter on the machine (but doesn’t solve the resources hogging issue). But I think that demonstrates pretty well why giving the power of scripts to just about anyone is really bad security-wise.

<scriptTask id="myScriptTask" scriptFormat="javascript">
    var s = new java.util.Scanner(java.lang.Runtime.getRuntime().exec("ifconfig").getInputStream()).useDelimiter("\\A");
    var output = s.hasNext() ? : "";
    java.lang.System.out.println("--- output = " + output);

Let’s deploy the process definition and execute a process instance:

public class Demo1 {

    public static void main (String[] args) {

        // Build engine and deploy
        ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
        RepositoryService repositoryService = processEngine.getRepositoryService();

        // Start process instance
        RuntimeService runtimeService = processEngine.getRuntimeService();

Which gives following output (shortened here):

— output = eth0 Link encap:Ethernet
inet addr: Bcast: Mask:

Process finished with exit code 1

It outputs information about all my network interfaces and then shutdows down the whole JVM. Yipes. That’s scary.

Trying Nashorn

The solution to our first problem is that we need to whitelist what we want to expose in a script, and have everything blacklisted by default. This way, users won’t be able to run any class or method that can do something malicious.

In Activiti, when a javascript script task is part of a process definition, we give this script to the javascript engine that is embedded in the JDK, using the ScriptEngine class in the JDK. In JDK 6/7 this was Rhino, in JDK 8 this is Nashorn. I first did some serious googling to find a solution for Nashorn (as this would be more future-proof). Nashorn does have a ‘class filter’ concept to effectively implement white-listing. However, the ScriptEngine abstraction does not have any facilities to actually tweak or configure the Nashorn engine. We’ll have to do some low-level magic to get it working.

Instead of using the default Nashorn scripting engine, we instantiate the Nashorn scripting engine ourselves in a ‘SecureScriptTask’ (which is a regular JavaDelegate). Note the use of the usage of jdk.nashorn.* package – not really nice. We follow the docs from to make the script execution more secure by adding a ‘ClassFilter’ to the Nashorn engine. This effectively acts as a white-list of approved classes that can be used in the script.

public class SafeScriptTaskDemo2 implements JavaDelegate {

    private Expression script;

    public void execute(DelegateExecution execution) throws Exception {
        NashornScriptEngineFactory factory = new NashornScriptEngineFactory();
        ScriptEngine scriptEngine = factory.getScriptEngine(new SafeClassFilter());

        ScriptingEngines scriptingEngines = Context

        Bindings bindings = scriptingEngines.getScriptBindingsFactory().createBindings(execution, false);
        scriptEngine.eval((String) script.getValue(execution), bindings);

        System.out.println("Java delegate done");

    public static class SafeClassFilter implements ClassFilter {

        public boolean exposeToScripts(String s) {
            return false;



When executed, the script above won’t be executed, an exception is thrown stating ‘Exception in thread “main” java.lang.RuntimeException: java.lang.ClassNotFoundException: java.lang.System.out.println’.

Note that the ClassFilter is only available from JDK 1.8.0_40 (quite recent!).

However, this doesn’t solve our second problem with infinite loops. Let’s execute a simple script:

while (true) {

You can guess what this’ll do. This will run forever. If you’re lucky, a transaction timeout will happen as the script task is executed in a transaction. But that’ far from a decent solution, as it hogs CPU resources for a while doing nothing.

The third problem, using a lot of memory, is also easy to demonstrate:

var array = []
for(var i = 0; i < 2147483647; ++i) {

When starting the process instance, the memory will quickly fill up (starting with only a couple of MB):

Screenshot from 2016-06-13 20:47:45

and eventually end with an OutOfMemoryException: Exception in thread “main” java.lang.OutOfMemoryError: GC overhead limit exceeded

Switching to Rhino

Between the following example and the previous one a lot of time was spent to make Nashorn somehow intercept or cope with the infinite loop/memory usage. However, after extensive searching and experimenting, it seems the features simply are not (yet?) in Nashorn. A quick search will teach you that we're not the only one looking for a solution to this. Often, it is mentioned that Rhino did have features on board to solve this.

For example in JDK < 8, the Rhino javascript engine had the 'instructionCount' callback mechanism, which is not present in Nashorn. It basically gives you a way to execute logic in a callback that is automatically called every x instructions (bytecode instructions!). I first tried (and lost a lot of time) to mimic the instructionCount idea with Nashorn, for example by prettifying the script first (because people could write the whole script on one line) and then injecting a line of code in the script that triggers a callback. However, that was 1) not very straightforward to do 2) one would still be able to write an instruction on one line that runs infinitely/uses a lot of memory.

Being stuck there, the search led us to the Rhino engine from Mozilla. Since its inclusion in the JDK a long time ago it actually evolved further on its own, while the version in the JDK wasn't updated with those changes! After reading up the (quite sparse) Rhino docs, it became clear Rhino seemed to have a far richer feature-set with regards to our use case. 

The ClassFilter from Nashorn matched the 'ClassShutter' concept in Rhino. The cpu and memory problem were solved using the callback mechanism of Rhino: you can define a callback that is called every x instructions. This means that one line could be hundreds of byte code instructions and we get a callback every x instructions .... which make it an excellent candidate for monitoring our cpu and memory usage when executing the script. 

If you are interested in our implementation of these ideas in the code, have a look here.

This does mean that whatever JDK version you are using, you will not be using the embedded javascript engine, but always Rhino.

Trying it out

To use the new secure scripting feature, add the following depdendency:


This will transitevly include the Rhino engine. This also enables the SecureJavascriptConfigurator, which needs to be configured before creating the process engine:

SecureJavascriptConfigurator configurator = new SecureJavascriptConfigurator()
  .setWhiteListedClasses(new HashSet<String>(Arrays.asList("java.util.ArrayList")))

ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration()

This will configure the secure scripting to

  • Every 10 instructions, check the CPU execution time and memory usage
  • Give the script 3 seconds and 3MB to execute
  • Limit stack depth to 10 (to avoid recursing)
  • Expose the array list as a class that is safe to use in the scripts

Running the script from above that tries to read the ifconfig and shut down the JVM leads to:

TypeError: Cannot call property getRuntime in object

[JavaPackage java.lang.Runtime]. It is not a function, it is "object".

Running the infinite loop script from above gives

Exception in thread “main” java.lang.Error: Maximum variableScope time of 3000 ms exceeded

And running the memory usage script from above gives

Exception in thread “main” java.lang.Error: Memory limit of 3145728 bytes reached

And hurray! The problems defined above are solved 🙂


I did a very unscientific quick check … and I almost didn’t dare to share it as the result go against what I assumed would happen. I created a quick main that runs a process instance with a script task 10000 times:

 public class PerformanceUnsecure { public static void main (String[] args) { ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); RepositoryService repositoryService = processEngine.getRepositoryService(); repositoryService.createDeployment().addClasspathResource("performance.bpmn20.xml").deploy(); Random random = new Random(); RuntimeService runtimeService = processEngine.getRuntimeService(); int nrOfRuns = 10000; long total = 0; for (int i=0; i<nrOfRuns; i++) { Map<String, Object> variables = new HashMap<String, Object>(); variables.put("a", random.nextInt()); variables.put("b", random.nextInt()); long start = System.currentTimeMillis(); runtimeService.startProcessInstanceByKey("myProcess", variables); long end = System.currentTimeMillis(); total += (end - start); } System.out.println("Finished process instances : " + processEngine.getHistoryService().createHistoricProcessInstanceQuery().count()); System.out.println("Total time = " + total + " ms"); System.out.println("Avg time/process instance = " + ((double)total/(double)nrOfRuns) + " ms"); } } 

The process definition is just a start -> script task -> end. The script task simply adds to variables and saves the result in a third variable.

<scriptTask id="myScriptTask" scriptFormat="javascript">
    var c = a + b;
    execution.setVariable('c', c);

I ran this five times, and got an average of 2.57 ms / process instance. This is on a recent JDK 8 (so Nashorn).

Then I switched the first couple of lines above to use the new secure scripting, thus switching to Rhino plus the security features enabled:

SecureJavascriptConfigurator configurator = new SecureJavascriptConfigurator()

ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration()

Did again five runs … and got 1.07 ms / process instance. Which is more than twice as fast for the same thing.

Of course, this is not a real test. I assumed the Rhino execution would be slower, with the class whitelisting checking and the callbacks … but no such thing. Maybe this particular case is one that is simply better suited for Rhino … If anyone can explain it, please leave a comment. But it is an interesting result nonetheless.


Activiti 5.21.0 Released

Activiti version 5.21.0 is released!

This release contains some quite important bugfixes, more specifically

Included in this release is the ‘secure scripting’ feature (for javascript), which is something many users were waiting for! Now you can execute javascript scripts in your process definitions without having to worry about harm that the script could cause. You can find the documentation in the ‘advanced’ section of the user guide. I will publish a blog about this secure scripting with more details one of the next days.

Thanks to the Activiti team for getting this release out, and a special thank you to all the people that contributed to this release: Martin Grofčík, Jan Petertonkoker, Kevin Schmidt, Gethin James, Zhouyan Ming, Vedran Pavić, Hasan Ramezani, daisuke-yoshimoto, Mariusz Olejnik, Roy Qu, Yannick Spillemaeckers and Jesper de Jong

Activiti 5.20.0 Released

Activiti 5.20.0 has been released. This is a bugfix release (mainly fixing the bug around losing message and signal start event subscriptions on process definition redeployment, see ACT-4117).

Get it from the download page or of course via Maven.

“Orchestrating Work with Activiti and Spring Integration” by Josh Long

When my buddy Josh Long from Pivotal writes about Activiti and Spring, it’s always must-read material. No exception this time, so I do want to give it exposure here too:

Orchestrating Work With Activiti and Spring Integration

Nice conclusion too: “BPM adds a fair amount of cognitive load for simple integrations but offers a lot of value when business process descriptions must be model- and business analyst-friendly. One common misconception is that developers have to give up the ability to enhance the system once BPM is involved; NOT SO! Thanks to Spring Boot and some hard work from the Activiti team, Activiti works perfectly with all of Spring.”


Edit 11 Feb 2016: as promised in the comments below, forked the example and made it running on v6:

Activiti released

Activiti has been released.

This is a bugfix release fixing following changes:

  • Security fix around using commons-collection (more info see the commit)
  • Upgrade to MyBatis 3.3.0
  • Enhancing AsyncExecutor for high load scenarios (especially when the job queue is full)
  • A few small bug fixes