Upcoming Webinar: Process Driven Spring Applications with Activiti – Sept 23rd

oss-logo-spring

Next week, I’ll be doing a webinar together with my friend Josh Long (he’s a Spring Developer Advocate, committer to many open source projects and of course Activiti). I will show some of the new Activiti tooling we’ve been working on recently, while Josh will demonstrate with live coding how easy it is to use Activiti in Spring Boot (spoiler: really easy).

You can register for the webinar for free here: https://spring.io/blog/2014/07/29/webinar-process-driven-spring-applications-with-activiti-sept-23rd

One day later, the Alfresco Summit will be kicked off in San Francisco. I’m joining two talks there:

For those who can’t make it to San Francisco: don’t worry, we’ll be landing in London two weeks later, Oct 7-9!

Seriously reducing memory consumption when running multiple Activiti Engines on the same JVM

The Activiti Forum is a place where I typically try to spend every day a bit of time. Surely, it is hard work to keep up with the influx of posts (but then again it means that Activiti is popular, so I’m not complaining!) but sometimes you find a true gem amongst the posts.

On the Forum, user jkronegg (or Julien, as he signs his posts) has posted an interesting finding which I’d like to give some more attention here. His use case is running multiple Activiti Process Engines on the same JVM. Maybe for multi-tenancy (one engine for each customer?), but that’s not the point here.

As you can read in his post, he has a setup of 50 Activiti Process Engines in the same JVM. And, while Activiti uses very little memory, he was interested if it could be brought down. More specifically, after profiling, he found that for every Activiti Process Engine most of the memory was consumed by the Mybatis internal Configuration object (Activiti uses Mybatis as database access framework). Conceptually, this configuration objects holds a parsed version of the Mybatis mapping files. As jkronegg correctly analyses, this data is actually static: two process engines configured the same will basically duplicate this Mybatis sql mapping information.

So, to verify this claim, I wrote a simple console application that you can find on Github. This little application gives you 3 options: start a number of ‘regular’ Activiti process engines, start a number of engines with a ‘shared mybatis config’ (I’ll come back to this) and a last option to check if you can actually use the engines (ie. a process deploys and runs).

01

So, time to boot up some ‘regular’ Activiti Engines, let’s say we boot up 250 regular Activiti Engines. I also have a profiler running (simply VisualVM, shipped with every JDK) to see the memory consumption:

02

You can clearly see that the memory nicely gets filled to about 1.5-1.6 GB (still only about 6MB for each engine).

When we calculate the objects with the biggest retained sizes, we can see that jkronegg is correct: the Configuration instances of Mybatis do take up 98.9% of all used memory!

03

Now, let’s see what happens if we apply the changes described in the forum post. More specifically, there are two main culprits that account for most of the memory usage (see the post for more details). This is proven when we inspect one of the instances: two objects (sqlFragments and mappedStatements) take up most of the memory of a Configuration instance:

04

By setting the reference on a new Mybatis Configuration object using reflection, the static data effectively gets shared between all next engines that get booted up. I did make a small change to the Activiti engine to make this easier in the future (with the current version you need to copy and paste the whole block of initSqlSessionFactory).

See SharedMyBatisProcessEngineConfiguration on Github for the implementation of the idea of copying the reference of those heavy objects (pretty much the same as described in the forum post).

If we now do choose option 2 in the console application, and boot up again 250 Activiti Engines, the memory  consumption looks as follows:

05

We’re not even touching the 250MB line (which would be 1MB / engine!) Only a fraction of the memory is being used, while still having exactly the same functionality!

and if we look at the number of instances and its retained size, we do indeed see there are 250 instances of the org.apache.ibatis.session.Configuration class, but now only account for 15,1% (was 98.9% before!)

06

Conclusion

So what can we learn about all of the above? Well, first of all, the reason why I posted this on my blog is because I simply love deep technical stuff like this. I’m not saying you should use this in production. You can be the judge of that yourself, it’s fiddling with reflection, depending on Mybatis internals and not tested beyond this simple example :-). But it is damn cool that you can quickly boot up so many Activiti engines with so little memory!

But what this really shows is the power of open source software. Both Activiti and Mybatis are open source and combine that with intelligent and creative developers you can get a whole lot further than with anything closed. Secondly this also demonstrates the awesomeness of the Activiti open source Community. There are a lot of people in our community, and by exchanging ideas and findings like above, Activiti gets better every day.

Thank you for that.

Running Activiti on JDK8

… just works!

All the Activiti code is written with JDK6 compatibility in mind, so it’s no surprise it just works out of the box.

Scripting

There are however some subtle changes when you use Script Task with javascript. In JDK 8, the Rhino engine has been replaced with Nashorn (which is a very good thing!).

nashorn

 

Our QA did show us that some of our unit tests with javascript script task did not pass on JDK 8. But don’t worry, the changes are minimal. More precise, we found two things

1. importing packages

In JDK 6/7 you can import a java package and then use classes from that package in your javascript (eg to use an arraylist):

importPackage (java.util);
var myVar = new ArrayList();

This doesn’t work on Nashorn and thus JDK 8. However, the nice developers of Nashorn have provided a way to support this in a backwards compatible way by loading a certain file:

load("nashorn:mozilla_compat.js");
importPackage (java.util);
var myVar = new ArrayList();

And everything will work again nicely.

2. Implicit variable fetching

This one is more subtle. On JDK 6/7 you could do this

var outcome = task.getVariableLocal('outcome');
if (outcome == 'approve') {
  task.setCategory('approved');
} else {
  task.setCategory('rejected');
}

However, due to an error in our unit test, we didn’t actually set the variable on local scope but on process instance scope. Still, the test worked on JDK 6/7 and failed on JDK8!

It took me some deep debugging, but finally I found the error. In JDK6/7, the getVariableLocal(‘outcome’) does resolve to null too. However, it does a fetch of getVariable(‘outcome’) immediately afterwards. This does not happen on JDK8.

There is a subtle thing going on here: on JDK6/7 something like

var outcome;

or

var outcome = null;

triggers a lookup in the ScriptingBindings, which defaults to a getVariable() call in Activiti. The script bindings call does not happen on JDK 8, which is actually more correct! So this is actually something to be aware of when using JDK6/7.

TL;DR

Activiti just works on JDK 8. If you are using javascript in your script task, be aware of subtle changes.



Review ‘Activiti 5.x Business Process Management’ by Zakir Laliwala and Irshad Mansuri

I’ve been contacted by the people of Packt Publishing to review their recent book release of the ‘Activiti 5.x Business Process Management”, written by Dr. Zakir Laliwala and Irshad Mansuri. For an open source project, books are a good thing. They indicate that a project is popular, and often people prefer books over gathering all the information from bits all over the internet. Don’t expect any insight into use cases, process modeling best practices or how to fit Activiti architecturally though, this is a hands-on description on how you get started technically with Activiti.

The book, as is clear from the cover, clearly aims to people who want to get started with Activiti and have no prior knowledge of it:

7065OS_Activiti 5.x Business Process Management Beginner's Guide

 

So, before going into more details, I’m happy to report that the book does exactly that. It introduces Activiti and its concepts gently and with a lot of screenshots of installing components but also when executing certain code. For people who are well advanced in Activiti certain things are a given, but I can imagine for people that have to start from zero these steps help to get bootstrapped quicker and with a lot less pain during learning.

In the past, I’ve read (and reviewed) books from Packt where the English wasn’t good to a point where it hindered reading it. However, the English in this book is really well-written and understandable. Absolutely no complaints here.

Some of the remarks I noted down some small remarks while reading the book:

  • The XML formatting of the process XML is often not readable due to lack of indentation and formatting. Probably a problem in general for printed books, but it’s very hard to understand the processes without structure.
  • I liked the fact that many concepts in the book were accompanied with a unit test, something I definitely support!
  • The bit about reporting had a bad screenshot, which kinda defeats the purpose of showing reporting
  • In chapter 5, it seems to hint that the activiti.cfg.xml way of configuring the engine is ‘Spring-only’, but actually this can be used outside of spring (only relying on spring-beans to have the parsing)
  • Chapter 6 about the services was really good: clearly explained and always with a short example/unit test. Also explains how to do debugging of processes. Many people will find this chapter useful
  • Chapter 7 was the same, but now for REST. Decent explanation of rest client install *and* unit testing with an http client. Nice.
  • Chapter 8 (integration with Liferay/Drools/OSGI) felt a bit ‘off’. It’s something I would have placed at the end of the book, and I definitely would have placed chapter 9 (advanced Activiti constructs) before as it fits better. Also, it doesn’t really go deep, it’s more about how Activiti is integrated with those technologies, accompanied with screenshots of the installation process. No insight into use cases or how it would be used in reality (granted, which is probably not the goal of this book).

So in summary: I cheer for everything being written about Activiti, so I really like the fact that this book was written. But more-so, I think it targets the beginners audience well. It has one goal: get a technical someone going with Activiti quickly and teaching the basics. I believe  people who are in the dark about Activiti and need to get started will be able to get going quickly with this book.

Important: Activiti 5.15 and MySQL 5.6+ users

Execute Custom queries in Activiti

(This will probably end up in the user guide of the Activiti 5.15 release, but I wanted to share it already)

The Activiti API allows for interacting with the database using a high level API. For example, for retrieving data the Query API and the Native Query API are powerful in its usage. However, for some use cases they might not be flexible enough. The following section described how a completely custom SQL statement (select, insert, updated and deletes are possible) can be executed against the Activiti data store, but completely within the configured Process Engine (and thus levering the transaction setup for example).

To define custom SQL statements, the Activiti engine leverages the capabilities of its underlying framework, MyBatis. The first thing to do when using custom SQL, is to create a MyBatis mapper class. More info can be read in the MyBatis user guide. For example, suppose that for some use case not the whole task data is needed, but only a small subset of it. A Mapper that could do this, looks as follows:

public interface MyTestMapper {

  @Select("SELECT ID_ as id, NAME_ as name, CREATE_TIME_ as createTime FROM ACT_RU_TASK")
  List<Map<String, Object>> selectTasks();

}

This mapper must be provided to the Process Engine configuration as follows:

...
<property name="customMybatisMappers">
  <set>
    <value>org.activiti.standalone.cfg.MyTestMapper</value>
  </set>
</property>
...

Notice that this is an interface. The underlying MyBatis framework will make an instance of it that can be used at runtime. Also notice that the return value of the method is not typed, but a list of maps (which corresponds to the list of rows with column values). Typing is possible with the MyBatis mappers if wanted.

To execute the query above, the managementService.executeCustomSql method must be used. This method takes in a CustomSqlExecution instance. This is a wrapper that hides the internal bits of the engine otherwise needed to make it work.

Unfortunately, Java generics make it a bit less readable than it could have been. The two generic types below are the mapper class and the return type class. However, the actual logic is simply to call the mapper method and return its results (if applicable).

CustomSqlExecution<MyTestMapper, List<Map<String, Object>>> customSqlExecution =
    new AbstractCustomSqlExecution<MyTestMapper, List<Map<String, Object>>>(MyTestMapper.class) {

  public List<Map<String, Object>> execute(MyTestMapper customMapper) {
    return customMapper.selectTasks();
  }

};

List<Map<String, Object>> results = managementService.executeCustomSql(customSqlExecution);

The Map entries in the list above will only contain id, name and create time in this case and not the full task object.

Any SQL is possible when using the approach above. Another more complex example:

  @Select({
    "SELECT task.ID_ as taskId, variable.LONG_ as variableValue FROM ACT_RU_VARIABLE variable",
    "inner join ACT_RU_TASK task on variable.TASK_ID_ = task.ID_",
    "where variable.NAME_ = #{variableName}"
  })
  List<Map<String, Object>> selectTaskWithSpecificVariable(String variableName);

Using this method, the task table will be joined with the variables table. Only where the variable has a certain name is retained, and the task id and the corresponding numerical value is returned.

This will be possible in Activiti 5.15. However, the code (and more specifically the Command implementation and the wrapper interface) can be used in any older version of Activiti.

Brazos For Activiti

As I mentioned a while ago, BP3 has become on of our enterprise partners for Activiti. The people of BP3 always said that they wanted to contribute to the open-source code of Activiti, but of course the proof of the pudding is in the eating.

We’re very enthousiast about what they’ve shown us recently: a full fledged drag and drop UI builder for Activiti forms, based on their experiences with building such a product on top IBM BPM (but now of course way cooler). I wanted to give it the exposure it deserves: see it for yourself!

And quoting Scott Francis,

we’re going to release this into the open source community on the same licensing terms as Activiti (Apache).  We think this could really elevate the way people approach building process-enabled application UIs with Activiti, and it seems like a great first contribution from BP3 to the Activiti open source community.

That’s just awesome. Stay tuned!

Advanced scripting in Activiti: Custom Configuration Injection

The scripting task is probably one of the ‘oldest’ classes in the Activiti code base, but I think it is still underused by many. The (perceived?) downsides are of course performance (interpretation vs compilation) and less support from IDE perspective.

However, the benefits (imo) outweigh this:

  • Scripts are defined in the process xml itself. No more worries about versioning and having to juggle with libs on the classpath.
  • What we’ve seen in the past is that less technical skilled people dare to try scripts. But never Java.

Anyway, what few people know or have realized is that you can do really awesome and advance stuff in scripts in Activiti. Since such a script is executed within the process engine, you have access to everything the engine is capable of. Yes … everything… which makes it both a very powerful but also (potential) dangerous thing (if you don’t know what you’re doing).

Let me walk you through such an example. I like to call it ‘custom configuration injection’ as a concept, because it effectively allows you to add custom logic at runtime which alters process execution significantly. If you have a cooler name for it, please let me know.

All code can be found on my Github Page: https://github.com/jbarrez/activiti-advanced-scripting

awesome-code-648x303

The use case

Now what is this thing I want to do. Well, I want to have a process that, when executed

  • Adds a ‘task completion event handler’ to every user task that is executed
  • This event handler must fire a custom event off to a remote URL, where potentially a event processor is doing its stuff

So basically, we want to fire off custom events to some remote URL whenever a task gets completed. A good use case for this could be Business Intelligence reporting/Complex event processing, eg with something like Esper.

Screen Shot 2013-07-23 at 10.03.11

The first version

The first cut of this functionality can be found at https://github.com/jbarrez/activit-advanced-scripting/blob/master/src/test/resources/org/activiti/test/my-process.bpmn20.xml. When this process is executed, the following happens:

var config = Context.getProcessEngineConfiguration();
var bpmnParser = config.getBpmnParser();

We simply fetch the current ProcessEngineConfiguration instance. We fetch the BpmnParser instance from this configuration, as we will want to change the general user task parsing for the whole engine.

Next, we build the script:

var script = "";
script = script + "importPackage(java.net);";
script = script + "importPackage(java.io);";
script = script + "var url = new URL('http://localhost:8182/echo');";
script = script + "var connection = url.openConnection();";
script = script + "connection.setRequestMethod('POST');";
script = script + "connection.setDoOutput(true);";
script = script + "var outputStream = new BufferedOutputStream(connection.getOutputStream());";
script = script + "outputStream.write(new java.lang.String(\"{'eventType':'task-complete'}\").bytes);";
script = script + "outputStream.flush();";
script = script + "connection.connect();";
script = script + "var respCode = connection.getResponseCode();";
script = script + "if (respCode != 200) ";
script = script + "println('Response code : ' + respCode);";
script = script + "outputStream.close();";
script = script + "connection.disconnect();";

This is obviously not the most efficient way to do this, but it sure shows the details of what happens. The message ‘eventType:task-complete’ is send to the localhost:8182 url through standard java.net and java.io classes.

The tricky part comes next:

var handler = new ExecuteScriptOnTaskCompleteBpmnParseHandler("JavaScript");
handler.setUserTaskCompleteScript(script);
bpmnParser.getBpmnParserHandlers().addHandler(handler);

// reset the deployment cache such that the new listener gets picked up on a new redeploy
config.getProcessDefinitionCache().clear();

Here we add a BpmnParseHandler class to the engine configuration. The parse handler will add the execution of the script defined above to every receival of the ‘task complete event’ send out by the engine. This parse handler kicks in every time a user task is parsed, which effectively adds our ‘send-event-to-remote-service’ to every user task now happening in your Activiti environment!

There is a unit test to see how this works: https://github.com/jbarrez/activiti-advanced-scripting/blob/master/src/test/java/org/activiti/test/ExecuteScriptInProcessTest.java. In the test, we setup a very simple ‘echo service’ which simply prints out whenever such an event is received. If you run it in your IDE, you’ll see something like this:

Screen Shot 2013-07-23 at 09.53.00

 

But we can do better

But we can do better. Check the following code.

var handler = new ExecuteScriptOnTaskCompleteBpmnParseHandler("JavaScript");
handler.setUserTaskCompleteScript("http://localhost:8182/scripts/task-complete.js");
handler.setExecuteScriptInJob(true);
bpmnParser.getBpmnParserHandlers().addHandler(handler);

// Update the configuration to use the correct job handler
var jobHandler = new ExecuteScriptJobHandler();
config.getJobHandlers().put(jobHandler.type,jobHandler);

This code does the same as in the previous section, ie. attaching a listener for ‘complete’ events to every user task. However, this implementation:

  • Executes the script asynchronously
  • Does not define the script in the process xml, but it is fetched from a remote url
  • Updates the job handler configuration

If you ask me, that’s pretty awesome! So this means that the actual sending of a message to the remote service is not impacting the execution performance of your process instance. Obviously, from here you can go crazy and add persistent queues and all that fanciness. And on top of that, the script is always fetched from a remote server. If you want to update the logic that is executed, simply change the script that is returned. This means you can impact process execution AT RUNTIME without touching the actual process.

There is a unit test for this at https://github.com/jbarrez/activiti-advanced-scripting/blob/master/src/test/java/org/activiti/test/ExecuteScriptWithJobTest.java

If you run this test,  you’ll see the following. Note that we host the completion script as static file called ‘task-complete.js’ on the test server.

Screen Shot 2013-07-23 at 09.50.36

In the test, you can see we have to execute the async job specifically to see the output of the test.

Caveat

On small caveat here: when the process engine reboots, the configuration will be reloaded from config file. Hence, the process from above that injects custom logic is not added. However, this can easily be done by using a ProcessEngineLifeCycleListener implementation that executes a process definition of a certain category after the engine has booted up. If you for example give all these processes ‘config-processes’ as category, they can easily be executed on bootup.

Conclusion

Scripting in BPMN 2.0 processes is a very powerful feature. It allows you to change process execution engine-wide in a matter of a few lines. Of course, all the code above can be done with Java. But the examples above use nothing more than standard BPMN 2.0 and the javascript engine that is bundled with every JDK install.

Thanks for reading. Happy coding!

NoSQL experimentations with Activiti: A (very simple) Neo4J prototype

logoI’ve got this itch for a long time now to try and see how easy it is to write a BPM engine on a graph database such as Neo4J. After all, the data model fits perfectly, as business processes are graphs and executions of those processes basically boil down to keeping pointers to where you are in that graph. It just feels as a very natural fit.

So I spend some time implementing a prototype, which you can find on https://github.com/jbarrez/activiti-neo4j

The prototype contains some unit tests which execute simple BPMN 2.0 processes. I tried to be as close as possible to the Activiti concepts of services, commands, executions, JavaDelegates, etc. Currently covered:

  • start and end event
  • user task
  • service task
  • parallel gateway
  • setting/getting variables

Much more information can be found at the readme on https://github.com/jbarrez/activiti-neo4j

I must say that working with Neo4j and its graph model for processes feel very very natural. It really simplifies stuff a lot and I can feel great potenital. Feel free to clone the code and check the code, it’s a regular maven project.

Note: we’re not moving to Neo4J with Activiti. This is just an experiment for myself. Of course, if you want to contribute you’re more than welcome!

Announcement: Enterprise Support for Activiti through BP3 and Edorasware

We often get questions (through email, forum or in person on conferences) where to go for enterprise support on Activiti. So far, we have not really been able to answer that question in an adequate way.

So we are *very* proud, happy and honored to be able to announce today that the Activiti project has selected two go-to partners who provide professional enterprise support for Activiti. Whether you’re in need for SLA backed services, expert consultancy or assistance on small to large-scale end-to-end Business Process Management projects, these are the people we trust and know you can build upon.

The partners in question are not of the least: BP3 and Edorasware. Both companies are deeply committed to the Activiti project and will have a major role to play in the vision and future of the Activiti project.

logo_bp3BP3 is a well-known company in the Business Process Management space. They operate from Austin, Texas (US), and have a vast expertise across multiple industries on deploying BPM projects. Scott Francis, the CTO of BP3 is a leading thought-leader and BPM blogger. Actually, Scott has blogged about Activiti since its first release and has followed and blogged about Activiti since then with great interest. Of course Scott is accompanied with a team of awesome people called the ‘BP3 Labs Group’ which will help driving the innovation in Activiti.

logo_edoraswareEdorasware, our other preferred go-to partner, is a prominent player in the BPM and ACM market. As a matter of fact, Edorasware was named a Gartner Cool Vendor for BPM just two weeks ago! On top of that, their core cloud and on-premise product called ‘Edoras One’ is built upon Activiti. Edorasware is a spin-off since 2010 of Mimacom and operates from Bern, Switzerland. Micha Kiener, CEO of Edorasware, and his team have been involved with the Activiti community since the first days of the project.

What does this mean for the Activiti project?

Nothing but good news, of course ;-)

It means that we now have an answer for people asking us where to go for Activiti enterprise support, or assistance on Activiti and more general, BPM projects.

Activiti of course remains 100% open source and community driven. Both BP3 and Edorasware will become an important part and driver in the open source community.

logo_alfresco

The main sponsor of Activiti, Alfresco, will keep investing in Activiti. And obviously, Alfresco keeps providing support for the Activiti usage within Alfresco when looking for Alfresco enterprise support.

For more information, see the Enterprise support Activiti website