Seriously reducing memory consumption when running multiple Activiti Engines on the same JVM

The Activiti Forum is a place where I typically try to spend every day a bit of time. Surely, it is hard work to keep up with the influx of posts (but then again it means that Activiti is popular, so I’m not complaining!) but sometimes you find a true gem amongst the posts.

On the Forum, user jkronegg (or Julien, as he signs his posts) has posted an interesting finding which I’d like to give some more attention here. His use case is running multiple Activiti Process Engines on the same JVM. Maybe for multi-tenancy (one engine for each customer?), but that’s not the point here.

As you can read in his post, he has a setup of 50 Activiti Process Engines in the same JVM. And, while Activiti uses very little memory, he was interested if it could be brought down. More specifically, after profiling, he found that for every Activiti Process Engine most of the memory was consumed by the Mybatis internal Configuration object (Activiti uses Mybatis as database access framework). Conceptually, this configuration objects holds a parsed version of the Mybatis mapping files. As jkronegg correctly analyses, this data is actually static: two process engines configured the same will basically duplicate this Mybatis sql mapping information.

So, to verify this claim, I wrote a simple console application that you can find on Github. This little application gives you 3 options: start a number of ‘regular’ Activiti process engines, start a number of engines with a ‘shared mybatis config’ (I’ll come back to this) and a last option to check if you can actually use the engines (ie. a process deploys and runs).

01

So, time to boot up some ‘regular’ Activiti Engines, let’s say we boot up 250 regular Activiti Engines. I also have a profiler running (simply VisualVM, shipped with every JDK) to see the memory consumption:

02

You can clearly see that the memory nicely gets filled to about 1.5-1.6 GB (still only about 6MB for each engine).

When we calculate the objects with the biggest retained sizes, we can see that jkronegg is correct: the Configuration instances of Mybatis do take up 98.9% of all used memory!

03

Now, let’s see what happens if we apply the changes described in the forum post. More specifically, there are two main culprits that account for most of the memory usage (see the post for more details). This is proven when we inspect one of the instances: two objects (sqlFragments and mappedStatements) take up most of the memory of a Configuration instance:

04

By setting the reference on a new Mybatis Configuration object using reflection, the static data effectively gets shared between all next engines that get booted up. I did make a small change to the Activiti engine to make this easier in the future (with the current version you need to copy and paste the whole block of initSqlSessionFactory).

See SharedMyBatisProcessEngineConfiguration on Github for the implementation of the idea of copying the reference of those heavy objects (pretty much the same as described in the forum post).

If we now do choose option 2 in the console application, and boot up again 250 Activiti Engines, the memory  consumption looks as follows:

05

We’re not even touching the 250MB line (which would be 1MB / engine!) Only a fraction of the memory is being used, while still having exactly the same functionality!

and if we look at the number of instances and its retained size, we do indeed see there are 250 instances of the org.apache.ibatis.session.Configuration class, but now only account for 15,1% (was 98.9% before!)

06

Conclusion

So what can we learn about all of the above? Well, first of all, the reason why I posted this on my blog is because I simply love deep technical stuff like this. I’m not saying you should use this in production. You can be the judge of that yourself, it’s fiddling with reflection, depending on Mybatis internals and not tested beyond this simple example :-). But it is damn cool that you can quickly boot up so many Activiti engines with so little memory!

But what this really shows is the power of open source software. Both Activiti and Mybatis are open source and combine that with intelligent and creative developers you can get a whole lot further than with anything closed. Secondly this also demonstrates the awesomeness of the Activiti open source Community. There are a lot of people in our community, and by exchanging ideas and findings like above, Activiti gets better every day.

Thank you for that.

2 Comments

  1. Justin May 11, 2016

    Will this work with activiti6 Beta3, I am using a extended version of activiti-app and sometimes see out of memory exceptions perm gen space … they are intermittant and thus far the cause remains ellusive. This product is about to go into production … you say that you personally consider activiti6 engine more stable … have you encountered any memory issues with it compared to activiti5

  2. Anonymous May 11, 2016

    @Justin: yes, the same principle applies, but I doubt it will solve your OOM exceptions … we’re talkin a few MB here. The best way to know what’s up is to attach a profiler to see what’s actually taking up the memory.

    I do consider v6 more stable, cause I know the foundations are better. However, it’s not as battle tested as v5. So given time, it’ll surpass v5 for sure.

Leave a Reply

Your email address will not be published. Required fields are marked *