How the Secure Scripting in Activiti works

One of the prominent features of the recent Activiti 5.21.0 release is ‘secure scripting’. The way to enable and use this feature is documented in detail in the Activiti user guide. In this post, I’ll show you how we came to its final implementation and what it’s doing under the hood. And of course, as it is my usual signature style, we’ll also have a bit of a look at the performance.

The Problem

The Activiti engine has supported scripting for script tasks (and task/execution listeners) since a long time. The scripts that are used are defined in the process definition and they can be executed directly after deploying the process definition. Which is something many people like. This is a big difference with Java delegate classes or delegate expressions, as they generally require putting the actual logic on the classpath. Which, in itself already introduces some sort of ‘protection’ as a power user generally only can do this. 

However, with scripts, no such ‘extra step’ is needed. If you give the power of script tasks to end users (and we know from some of our users some companies do have this use case), all bets are pretty much off. You can shut down the JVM or do malicious things by executing a process instance.

A second problem is that it’s quite easy to write a script that does an infinite loop and never ends. A third problem is that a script can easily use a lot of memory when executed and hog a lot of system resources.

Let’s look at the first problem for starters. First off all, let’s add the latest and greatest Activiti engine dependency and the H2 in memory database library:

<dependencies>
  <dependency>
    <groupId>org.activiti</groupId>
    <artifactId>activiti-engine</artifactId>
    <version>5.21.0</version>
  </dependency>
  <dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <version>1.3.176</version>
  </dependency>
</dependencies>

The process we’ll use here is trivially simple: just a start event, script task and end. The process is not really the point here, the script execution is. Screenshot from 2016-06-13 20:16:21

The first script we’ll try does two things: it will get and display my machine’s current network configuration (but there are obviously more dangerous applications of this idea) and then shutdown the whole JVM. Of course, in a proper setup, some of this will be mitigated by making sure that the user running the logic does not have any rights that matter on the machine (but doesn’t solve the resources hogging issue). But I think that demonstrates pretty well why giving the power of scripts to just about anyone is really bad security-wise.

<scriptTask id="myScriptTask" scriptFormat="javascript">
  <script>
    var s = new java.util.Scanner(java.lang.Runtime.getRuntime().exec("ifconfig").getInputStream()).useDelimiter("\\A");
    var output = s.hasNext() ? s.next() : "";
    java.lang.System.out.println("--- output = " + output);
    java.lang.System.exit(1);
  </script>
</scriptTask>

Let’s deploy the process definition and execute a process instance:

public class Demo1 {

    public static void main (String[] args) {

        // Build engine and deploy
        ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
        RepositoryService repositoryService = processEngine.getRepositoryService();
        repositoryService.createDeployment().addClasspathResource("process.bpmn20.xml").deploy();

        // Start process instance
        RuntimeService runtimeService = processEngine.getRuntimeService();
        runtimeService.startProcessInstanceByKey("myProcess");
    }
}

Which gives following output (shortened here):

— output = eth0 Link encap:Ethernet
inet addr:192.168.0.114 Bcast:192.168.0.255 Mask:255.255.255.0

Process finished with exit code 1

It outputs information about all my network interfaces and then shutdows down the whole JVM. Yipes. That’s scary.

Trying Nashorn

The solution to our first problem is that we need to whitelist what we want to expose in a script, and have everything blacklisted by default. This way, users won’t be able to run any class or method that can do something malicious.

In Activiti, when a javascript script task is part of a process definition, we give this script to the javascript engine that is embedded in the JDK, using the ScriptEngine class in the JDK. In JDK 6/7 this was Rhino, in JDK 8 this is Nashorn. I first did some serious googling to find a solution for Nashorn (as this would be more future-proof). Nashorn does have a ‘class filter’ concept to effectively implement white-listing. However, the ScriptEngine abstraction does not have any facilities to actually tweak or configure the Nashorn engine. We’ll have to do some low-level magic to get it working.

Instead of using the default Nashorn scripting engine, we instantiate the Nashorn scripting engine ourselves in a ‘SecureScriptTask’ (which is a regular JavaDelegate). Note the use of the usage of jdk.nashorn.* package – not really nice. We follow the docs from https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/api.html to make the script execution more secure by adding a ‘ClassFilter’ to the Nashorn engine. This effectively acts as a white-list of approved classes that can be used in the script.

public class SafeScriptTaskDemo2 implements JavaDelegate {

    private Expression script;

    public void execute(DelegateExecution execution) throws Exception {
        NashornScriptEngineFactory factory = new NashornScriptEngineFactory();
        ScriptEngine scriptEngine = factory.getScriptEngine(new SafeClassFilter());

        ScriptingEngines scriptingEngines = Context
                .getProcessEngineConfiguration()
                .getScriptingEngines();

        Bindings bindings = scriptingEngines.getScriptBindingsFactory().createBindings(execution, false);
        scriptEngine.eval((String) script.getValue(execution), bindings);

        System.out.println("Java delegate done");
    }

    public static class SafeClassFilter implements ClassFilter {

        public boolean exposeToScripts(String s) {
            return false;
        }

    }

}

When executed, the script above won’t be executed, an exception is thrown stating ‘Exception in thread “main” java.lang.RuntimeException: java.lang.ClassNotFoundException: java.lang.System.out.println’.

Note that the ClassFilter is only available from JDK 1.8.0_40 (quite recent!).

However, this doesn’t solve our second problem with infinite loops. Let’s execute a simple script:

while (true) {
  print("Hello");
}

You can guess what this’ll do. This will run forever. If you’re lucky, a transaction timeout will happen as the script task is executed in a transaction. But that’ far from a decent solution, as it hogs CPU resources for a while doing nothing.

The third problem, using a lot of memory, is also easy to demonstrate:

var array = []
for(var i = 0; i < 2147483647; ++i) {
  array.push(i);
  java.lang.System.out.println(array.length);
}

When starting the process instance, the memory will quickly fill up (starting with only a couple of MB):

Screenshot from 2016-06-13 20:47:45

and eventually end with an OutOfMemoryException: Exception in thread “main” java.lang.OutOfMemoryError: GC overhead limit exceeded

Switching to Rhino

Between the following example and the previous one a lot of time was spent to make Nashorn somehow intercept or cope with the infinite loop/memory usage. However, after extensive searching and experimenting, it seems the features simply are not (yet?) in Nashorn. A quick search will teach you that we’re not the only one looking for a solution to this. Often, it is mentioned that Rhino did have features on board to solve this.

For example in JDK < 8, the Rhino javascript engine had the 'instructionCount' callback mechanism, which is not present in Nashorn. It basically gives you a way to execute logic in a callback that is automatically called every x instructions (bytecode instructions!). I first tried (and lost a lot of time) to mimic the instructionCount idea with Nashorn, for example by prettifying the script first (because people could write the whole script on one line) and then injecting a line of code in the script that triggers a callback. However, that was 1) not very straightforward to do 2) one would still be able to write an instruction on one line that runs infinitely/uses a lot of memory.

Being stuck there, the search led us to the Rhino engine from Mozilla. Since its inclusion in the JDK a long time ago it actually evolved further on its own, while the version in the JDK wasn’t updated with those changes! After reading up the (quite sparse) Rhino docs, it became clear Rhino seemed to have a far richer feature-set with regards to our use case.

The ClassFilter from Nashorn matched the ‘ClassShutter’ concept in Rhino. The cpu and memory problem were solved using the callback mechanism of Rhino: you can define a callback that is called every x instructions. This means that one line could be hundreds of byte code instructions and we get a callback every x instructions …. which make it an excellent candidate for monitoring our cpu and memory usage when executing the script.

If you are interested in our implementation of these ideas in the code, have a look here.

This does mean that whatever JDK version you are using, you will not be using the embedded javascript engine, but always Rhino.

Trying it out

To use the new secure scripting feature, add the following depdendency:

<dependency>
  <groupId>org.activiti</groupId>
  <artifactId>activiti-secure-javascript</artifactId>
  <version>5.21.0</version>
</dependency>

This will transitevly include the Rhino engine. This also enables the SecureJavascriptConfigurator, which needs to be configured before creating the process engine:

SecureJavascriptConfigurator configurator = new SecureJavascriptConfigurator()
  .setWhiteListedClasses(new HashSet<String>(Arrays.asList("java.util.ArrayList")))
  .setMaxStackDepth(10)
  .setMaxScriptExecutionTime(3000L)
  .setMaxMemoryUsed(3145728L)
  .setNrOfInstructionsBeforeStateCheckCallback(10);

ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration()
  .addConfigurator(configurator)
  .buildProcessEngine();

This will configure the secure scripting to

  • Every 10 instructions, check the CPU execution time and memory usage
  • Give the script 3 seconds and 3MB to execute
  • Limit stack depth to 10 (to avoid recursing)
  • Expose the array list as a class that is safe to use in the scripts

Running the script from above that tries to read the ifconfig and shut down the JVM leads to:

TypeError: Cannot call property getRuntime in object [JavaPackage java.lang.Runtime]. It is not a function, it is “object”.

Running the infinite loop script from above gives

Exception in thread “main” java.lang.Error: Maximum variableScope time of 3000 ms exceeded

And running the memory usage script from above gives

Exception in thread “main” java.lang.Error: Memory limit of 3145728 bytes reached

And hurray! The problems defined above are solved 🙂

Performance

I did a very unscientific quick check … and I almost didn’t dare to share it as the result go against what I assumed would happen.

I created a quick main that runs a process instance with a script task 10000 times:

public class PerformanceUnsecure {

    public static void main (String[] args) {

        ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

        RepositoryService repositoryService = processEngine.getRepositoryService();
        repositoryService.createDeployment().addClasspathResource("performance.bpmn20.xml").deploy();

        Random random = new Random();

        RuntimeService runtimeService = processEngine.getRuntimeService();

        int nrOfRuns = 10000;
        long total = 0;

        for (int i=0; i<nrOfRuns; i++) {
            Map<String, Object> variables = new HashMap<String, Object>();
            variables.put("a", random.nextInt());
            variables.put("b", random.nextInt());
            long start = System.currentTimeMillis();
            runtimeService.startProcessInstanceByKey("myProcess", variables);
            long end = System.currentTimeMillis();
            total += (end - start);
        }
        System.out.println("Finished process instances : " + processEngine.getHistoryService().createHistoricProcessInstanceQuery().count());
        System.out.println("Total time = " + total + " ms");
        System.out.println("Avg time/process instance = " + ((double)total/(double)nrOfRuns) + " ms");
    }

}

The process definition is just a start -> script task -> end. The script task simply adds to variables and saves the result in a third variable.

<scriptTask id="myScriptTask" scriptFormat="javascript">
  <script>
    var c = a + b;
    execution.setVariable('c', c);
  </script>
</scriptTask>

I ran this five times, and got an average of 2.57 ms / process instance. This is on a recent JDK 8 (so Nashorn).

Then I switched the first couple of lines above to use the new secure scripting, thus switching to Rhino plus the security features enabled:

SecureJavascriptConfigurator configurator = new SecureJavascriptConfigurator()
  .addWhiteListedClass("org.activiti.engine.impl.persistence.entity.ExecutionEntity")
  .setMaxStackDepth(10)
  .setMaxScriptExecutionTime(3000L)
  .setMaxMemoryUsed(3145728L)
  .setNrOfInstructionsBeforeStateCheckCallback(1);

ProcessEngine processEngine = new StandaloneInMemProcessEngineConfiguration()
  .addConfigurator(configurator)
  .buildProcessEngine();

Did again five runs … and got 1.07 ms / process instance. Which is more than twice as fast for the same thing.

Of course, this is not a real test. I assumed the Rhino execution would be slower, with the class whitelisting checking and the callbacks … but no such thing. Maybe this particular case is one that is simply better suited for Rhino … If anyone can explain it, please leave a comment. But it is an interesting result nonetheless.

Conclusion

If you are using scripts in your process definition, do read up on this new secure scripting feature in the engine. As this is a new feature, feedback and improvements are more than welcome!

Activiti 5.21.0 Released

We just released Activiti version 5.21.0!

This release contains some quite important bugfixes, more specifically

Included in this release is the ‘secure scripting’ feature (for javascript), which is something many users were waiting for! Now you can execute javascript scripts in your process definitions without having to worry about harm that the script could cause. You can find the documentation in the ‘advanced’ section of the user guide. I will publish a blog about this secure scripting with more details one of the next days.

Thanks to the Activiti team for getting this release out, and a special thank you to all the people that contributed to this release: Martin Grofčík, Jan Petertonkoker, Kevin Schmidt, Gethin James, Zhouyan Ming, Vedran Pavić, Hasan Ramezani, daisuke-yoshimoto, Mariusz Olejnik, Roy Qu, Yannick Spillemaeckers and Jesper de Jong

Activiti 5.20.0 Released

We’ve just released Activiti 5.20.0. This is a bugfix release (mainly fixing the bug around losing message and signal start event subscriptions on process definition redeployment, see ACT-4117).

Get it from the download page or of course via Maven.

“Orchestrating Work with Activiti and Spring Integration” by Josh Long

When my buddy Josh Long from Pivotal writes about Activiti and Spring, it’s always must-read material. No exception this time, so I do want to give it exposure here too:

Orchestrating Work With Activiti and Spring Integration

Nice conclusion too: “BPM adds a fair amount of cognitive load for simple integrations but offers a lot of value when business process descriptions must be model- and business analyst-friendly. One common misconception is that developers have to give up the ability to enhance the system once BPM is involved; NOT SO! Thanks to Spring Boot and some hard work from the Activiti team, Activiti works perfectly with all of Spring.”

 

Edit 11 Feb 2016: as promised in the comments below, forked the example and made it running on v6: https://github.com/jbarrez/activiti-examples

Activiti 5.19.0.2 released

We just released Activiti 5.19.0.2.

This is a bugfix release fixing following changes:

  • Security fix around using commons-collection (more info see the commit)
  • Upgrade to MyBatis 3.3.0
  • Enhancing AsyncExecutor for high load scenarios (especially when the job queue is full)
  • A few small bug fixes

 

Activiti Community Meetup Antwerp – March 2nd 2016 !

It’s been way too long ago we’ve had an Activiti meetup in Belgium (which is where I live, so there is no excuse really). Over a lunch with my good friend Yannick Spillemaeckers, we decided to rectify that.

So I’m proud to announce we’ll be having a meetup March 2nd 2016 in Antwerp! You can find all information about the event here: https://www.eventbrite.com/e/activiti-community-meetup-antwerp-tickets-20915929107

Please do register on that Eventbrite page, seats are limited and full is really full! We’ve gone for an evening event, so that people can join after work hours and enjoy and evening of talking about Activiti with like-minded peers (a version 6 presentation by yours truly, amongst others). And of course, attending it is completely free.

For now (see the Eventbrite page), we have two sessions, but we’re still in planning mode. Check the page in the near future for updates …but do reserve your seat today! First come, first serve.

Activiti 6.0.0.Beta2 released

Just in time for Christmas, we’ve released the second beta of Activiti v6! You can get it through the usual channels: the activiti.org website or through Maven.

This release is mainly a ‘refactoring-release’ that prepares the code for all the nifty stuff we want to do next: low in new features, but the internal kitchen has seen some major changes since beta1. We’ve been contacted by a few companies that are already giving the new pluggable persistence a go, and the feedback has been very positive.

In the new year, we’ll be ramping up the beta releases and add the ‘v6 specific features’, that will show off the huge work we did from v5 to v6 and make the upgrade worth the while. Version 6 is already feature-par with v5 (and as stable as v5). We’ll also start focussing on performance: we know that especially the simple cases probably are a tad slower (due to the richer execution tree, which means a few more row inserts), but likewise we’ve seen some complex examples being way more simpler and performant (due to the smarter execution tree). But all of that of course will be posted online in an open and reproducible way, as we always do! At the same time, we’ll ramp up on the documentation side of things: way more than in v5 we can now clearly document examples and how this translates to the internal execution.

Here’s the release highlights:

  • We’ve seriously refactored the persistence logic of the engine, making the persistence layer completely pluggable! Read more about it here.
  • Refactored the BpmnDeployer class and related logic to allow for easier pluggability/extensibility.
  • Refactored the internal execution tree structure when using multi instance behavior: there will now always be one execution to demarcate the ‘scope’ of the multi instance. This opens up future use cases (storing variables on that scope only, terminating only one instance or all, etc.)
  • Extended the ‘terminate all event’ behaviour to cover more use cases and be fully consistent with the BPMN spec. Also,
  • Added the DynamicBpmnService. Read more about it here.
  • Various and many small bugfixes all around.

All feedback of course, more than welcome!

Merry Christmas to all!

 

Activiti on start.spring.io !

In case you missed the following tweet last week

 

That’s right! Activiti is now on start.spring.io!

This is a huge deal – the Spring Initializr is the place where the journey for many Spring Boot projects start, so being on that page is a huge honour for us. Of course, couldn’t have done it without the help of my Spring friends Josh Long (who also contributed the Spring Boot + Activiti code) and Stéphane Nicoll.

And just to prove how awesome it is, here’s a short movie that starts on start.spring.io and finishes with a REST endpoint that starts a process instance is about 5 minutes!

Achievement Unlocked: 5000 Forum Posts!

aunlocked

I just passed the 5000 forum posts on the Activiti Forum. Out of a total of 41792 posts, so you can only blame me for  12% of the content on there ;-).

Just thinking about it humbles me tremendously. The forum is such an important tool for improving the Activiti Engine, guarding the quality and getting fresh ideas for features.

Also, doing the math that averages to 2.5 posts/day (not taking in account any weekends and holidays). Crazy! Thank you for all the conversations over the past years!

Multi-Tenancy with separate database schemas in Activiti

One feature request we’ve heard in the past is that of running the Activiti engine in a multi-tenant way where the data of a tenant is isolated from the others. Certainly in certain cloud/SaaS environments this is a must.

A couple of months ago I was approached by Raphael Gielen, who is a student at the university of Bonn, working on a master thesis about multi-tenancy in Activiti. We got together in a co-working coffee bar a couple of weeks ago, bounced ideas and hacked together a first prototype with database schema isolation for tenants. Very fun :-).

Anyway, we’ve been refining and polishing that code and committed it to the Activiti codebase. Let’s have a look at the existing ways of doing multi-tenancy with Activiti in the first two sections below. In the third section, we’ll dive into the new multi-tenant multi-schema feature sprinkled with some real-working code examples!

Shared Database Multi-tenancy

Activiti has been multi-tenant capable for a while now (since version 5.15). The approach taken was that of a shared database: there is one (or more) Activiti engines and they all go to the same database. Each entry in the database table has a tenant identifier, which is best to be understood as a sort of tag for that data. The Activiti engine and API’s then read and use that tenant identifier to perform its various operations in the context of a tenant.

For example, as shown in the picture below, two different tenants can have a process definition with the same key. The engine and API’s make sure there is no mixup of data.

Screenshot 2015-10-06 12.57.00

The benefit of this approach is the simplicity of deployment, as there is no difference from setting up a ‘regular’ Activiti engine. The downside is that you have to remember to use the right API calls (i.e. those that take in account the tenant identifier). Also, it has the same problem as any system with shared resources: there will always be competition for the resources between tenants. In most use cases this is fine, but there are use cases that can’t be done in this way, like giving certain tenants more or less system resources.

Multi-Engine Multi-Tenancy

Another approach, which has been possible since the very first version of Activiti is simply having one engine instance for each tenant:

Screenshot 2015-10-06 13.12.56

In this setup, each tenant can have different resource configurations or even run on different physical servers. Each engine in this picture here can of course be multiple engines for more performance/failover/etc. The benefit now is that the resources are tailored for the tenant. The downside is the more complex setup (multiple database schemas, having a different configuration file for each tenant, etc.). Each engine instance will take up memory (but that’s very low with Activiti). Also, you’d need t write some routing component that knows somehow the current tenant context and routes to the correct engine.

Screenshot 2015-10-06 13.18.36

Multi-Schema Multi-Tenancy

The latest addition to the Activiti multi-tenancy story was added two weeks ago (here’s the commit), simultaneously on version 5 and 6. Here, there is a database (schema) for each tenant, but only one engine instance. Again, in practice there might be multiple instances for performance/failover/etc., but the concept is the same:

Screenshot 2015-10-06 13.41.20

The benefit is obvious: there is but one engine instance to manage and configure and the API’s are exactly the same as with a non-multi-tenant engine. But foremost, the data of a tenant is completely separated from the data of other tenants. The downside (similar to the multi-engine multi-tenant approach) is that someone needs to manage and configure different databases. But the complex engine management is gone.

The commit I linked to above also contains a unit test showing how the Multi-Schema Multi-Tenant engine works.

Building the process engine is easy, as there is a MultiSchemaMultiTenantProcessEngineConfiguration that abstracts away most of the details:


config = new MultiSchemaMultiTenantProcessEngineConfiguration(tenantInfoHolder);

config.setDatabaseType(MultiSchemaMultiTenantProcessEngineConfiguration.DATABASE_TYPE_H2);
config.setDatabaseSchemaUpdate(MultiSchemaMultiTenantProcessEngineConfiguration.DB_SCHEMA_UPDATE_DROP_CREATE);
    
config.registerTenant("alfresco", createDataSource("jdbc:h2:mem:activiti-mt-alfresco;DB_CLOSE_DELAY=1000", "sa", ""));
config.registerTenant("acme", createDataSource("jdbc:h2:mem:activiti-mt-acme;DB_CLOSE_DELAY=1000", "sa", ""));
config.registerTenant("starkindustries", createDataSource("jdbc:h2:mem:activiti-mt-stark;DB_CLOSE_DELAY=1000", "sa", ""));
    
processEngine = config.buildProcessEngine();

This looks quite similar to booting up a regular Activiti process engine instance. The main difference is that we’re registring tenants with the engine. Each tenant needs to be added with its unique tenant identifier and Datasource implementation. The datasource implementation of course needs to have its own connection pooling. This means you can effectively give certain tenants different connection pool configuration depending on their use case. The Activiti engine will make sure each database schema has been either created or validated to be correct.

The magic to make this all work is the TenantAwareDataSourceThis is a javax.sql.DataSource implementation that delegates to the correct datasource depending on the current tenant identifier. The idea of this class was heavily influenced by Spring’s AbstractRoutingDataSource (standing on the shoulders of other open-source projects!).

The routing to the correct datasource is being done by getting the current tenant identifier from the TenantInfoHolder instance. As you can see in the code snippet above, this is also a mandatory argument when constructing a MultiSchemaMultiTenantProcessEngineConfiguration. The TenantInfoHolder is an interface you need to implement, depending on how users and tenants are managed in your environment. Typically you’d use a ThreadLocal to store the current user/tenant information (much like Spring Security does) that gets filled by some security filter. This class effectively acts as the routing component’ in the picture below:

Screenshot 2015-10-06 13.53.13

In the unit test example, we use indeed a ThreadLocal to store the current tenant identifier, and fill it up with some demo data:

 private void setupTenantInfoHolder() {
    DummyTenantInfoHolder tenantInfoHolder = new DummyTenantInfoHolder();
    
    tenantInfoHolder.addTenant("alfresco");
    tenantInfoHolder.addUser("alfresco", "joram");
    tenantInfoHolder.addUser("alfresco", "tijs");
    tenantInfoHolder.addUser("alfresco", "paul");
    tenantInfoHolder.addUser("alfresco", "yvo");
    
    tenantInfoHolder.addTenant("acme");
    tenantInfoHolder.addUser("acme", "raphael");
    tenantInfoHolder.addUser("acme", "john");
    
    tenantInfoHolder.addTenant("starkindustries");
    tenantInfoHolder.addUser("starkindustries", "tony");
    
    this.tenantInfoHolder = tenantInfoHolder;
  }
 

We now start some process instance, while also switching the current tenant identifier. In practice, you have to imagine that multiple threads come in with requests, and they’ll set the current tenant identifier based on the logged in user:

startProcessInstances("joram");
startProcessInstances("joram");
startProcessInstances("raphael");
completeTasks("raphael");

The startProcessInstances method above will set the current user and tenant identifier and start a few process instances, using the standard Activiti API as if there was no multi tenancy at all (the completeTasks method similarly completes a few tasks).

Also pretty cool is that you can dynamically register (and delete) new tenants, by using the same method that was used when building the process engine. The Activiti engine will make sure the database schema is either created or validated.

config.registerTenant("dailyplanet", createDataSource("jdbc:h2:mem:activiti-mt-daily;DB_CLOSE_DELAY=1000", "sa", ""));

Here’s a movie showing the unit test being run and the data effectively being isolated:

Multi-Tenant Job Executor

The last piece to the puzzle is the job executor. Regular Activiti API calls ‘borrow’ the current thread to execute its operations and thus can use any user/tenant context that has been set before on the thread.

The job executor however, runs using a background threadpool and has no such context. Since the AsyncExecutor in Activiti is an interface, it isn’t hard to implement a multi-schema multi-tenant job executor. Currently, we’ve added two implementations. The first implementation is called the SharedExecutorServiceAsyncExecutor:

config.setAsyncExecutorEnabled(true);
config.setAsyncExecutorActivate(true);
config.setAsyncExecutor(new SharedExecutorServiceAsyncExecutor(tenantInfoHolder));

This implementations (as the name implies) uses one threadpool for all tenants. Each tenant does have its own job acquisition threads, but once the job is acquired, it is put on the shared threadpool. The benefit of this system is that the number of threads being used by Activiti is constrained.

The second implementation is called the ExecutorPerTenantAsyncExecutor:

config.setAsyncExecutorEnabled(true);
config.setAsyncExecutorActivate(true);
config.setAsyncExecutor(new ExecutorPerTenantAsyncExecutor(tenantInfoHolder));

As the name implies, this class acts as a ‘proxy’ AsyncExecutor. For each tenant registered, a complete default AsyncExecutor is booted. Each with its own acquisition threads and execution threadpool. The ‘proxy’ simply delegates to the right AsyncExecutor instance. The benefit of this approach is that each tenant can have a fine-grained job executor configuration, tailored towards the needs of the tenant.

Conclusion

As always, all feedback is more than welcome. Do give the multi-schema multi-tenancy a go and let us know what you think and what could be improved for the future!