In the Activiti 5.12 release, we added reporting capabilities on top of the Activiti engine, demonstrating the concepts through the Activiti Explorer web application (but of course usable everywhere).

Now, don’t be fooled: since a very long time , the Activiti engine has the capability of gathering historical or audit data when you execute business processes. All this data is stored in the historical database tables and can thus be easily queried. Which means that any reporting tool such as JasperReports, Birt, Crystal Reports etc. can just use the historical tables as a datasource to produce reports in any format you’d like (Word, PDF, …) to get insight how your business is executing its business processes. I’ll probably blog such an example pretty soon.

Eating our own dogfood

But the thing where I’d like to focus on today is the web side of things: web charts/reports which can be combined into a dashboard for example. The first thing we must be able to do is to expose the historical data in a way we can use it to create these charts. But where do you put the logic (the queries and data manipulation) to generate a dataset for the chart and/or the report? Do you embed the SQL in your UI-layer? Of course not. What if multiple applications want to use the data? What if we want to store the generated dataset to get a snapshot of the data at a certain point in time.

When we thought about this problem we first though about it in the traditional way. A new service with reporting capabilities, probably using some sort of DSL to define the dataset generation which are stored in some kind of data store. Anyway, a whole new things and concepts to learn and master. Not to mention extra implementation and maintenance.

But then it hit us. Everything we needed is already available in the Activiti engine. If we use a process to define the logic to create the dataset for the report, we can leverage all the facilities of the engine. The only requirement for this process is that it generates the JSON data which follows a fixed format. Some benefits

• The process has straight access to the internals of the Activiti engine. It has direct access to the database used by the engine.
• The dataset that is generated can be stored in the historical tables of Activiti if wanted. So we have a ‘save report data’ mechanism for free.
• The job executor can be used as for any other process. This means that you can asynchronously generate the process or only execute certain steps asynchronously. It also means you can use timers, eg. to generate the report data on certain points in time.
• Creating a new report can be done with known tools and known concepts. Also, no new concepts, services or applications are needed. Deploying or uploading a new report is the same as deploying a new process. Generating a report is the same as running a process instance.
• It allows to use the BPMN 2.0 constructs. This means that all things like parallel steps, do branching based on data or even request user input during the generation are possible out-of-the-box.

A dash of Javascript

Since the generation of the dataset is done by a process, everything possible in a process can be used. So you can use Java delegate classes or whatever you fancy.

But since the kool kids nowadays are using Javascript, we added some example process to the demo data of Activiti Explorer that use the scripting functionality in the engine. The nice thing about Javascript is that JSON is native to it, and creating a JSON object is really easy. As said above, the only requirement for such a process is that it must generate the JSON dataset following the predefined format.

For example to generate an overview of all past process instances we could have a process like this:

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:activiti="http://activiti.org/bpmn"
 xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:omgdc="http://www.omg.org/spec/DD/20100524/DC"
 xmlns:omgdi="http://www.omg.org/spec/DD/20100524/DI" typeLanguage="http://www.w3.org/2001/XMLSchema"
 expressionLanguage="http://www.w3.org/1999/XPath"
 targetNamespace="activiti-report">

<process id="process-instance-overview-report" name="Process Instance Overview" isExecutable="true">

 <startEvent id="startevent1" name="Start" />
 <sequenceFlow id="flow1" sourceRef="startevent1" targetRef="generateDataset" />

 <scriptTask id="generateDataset" name="Execute script" scriptFormat="JavaScript" activiti:autoStoreVariables="false">
 <script><![CDATA[

  importPackage(java.sql);
  importPackage(java.lang);
  importPackage(org.activiti.explorer.reporting);

  var result = ReportingUtil.executeSelectSqlQuery("SELECT PD.NAME_, PD.VERSION_ , count(*) FROM ACT_HI_PROCINST PI
       inner join ACT_RE_PROCDEF PD on PI.PROC_DEF_ID_ = PD.ID_ group by PROC_DEF_ID_");

  var reportData = {};
  reportData.datasets = [];
  
  // Native json usage
  var dataset = {};
  dataset.type = "pieChart";
  dataset.description = "Process instance overview (" + new java.util.Date() + ")";
  dataset.data = {};

  while (result.next()) { // process results one row at a time
    var name = result.getString(1);
    var version = result.getLong(2)
    var count = result.getLong(3);
    dataset.data[name + " (v" + version + ")"] = count;
  }
  reportData.datasets.push(dataset);

  // Storing the json as process variable
  execution.setVariable("reportData", new java.lang.String(JSON.stringify(reportData)).getBytes("UTF-8"));
 ]]></script>
 </scriptTask>
 <sequenceFlow id="flow3" sourceRef="generateDataset" targetRef="theEnd" />

 <endEvent id="theEnd" />

 </process></em>

</definitions>

The script is pretty easy to understand. All it does is querying the database, creating a json object with the data and storing it as process variable. Since it is stored as a variable, it is basically a snapshot of the dataset. So at any time in the future you can just fetch the json data and look at the report as it was at that point in time.

The json produced by the process above can then easily be used to generate charts, as demonstrated by the Activiti Explorer application:

It is also very easy to see how easy it is to create a dashboard app with the same approach. But that’ll be for a next blogpost.

Thanks for reading!

Introduction

Generally, the Activiti engine works in a stateless way. All the state about process instances, tasks and whatnot is stored in the database and only retrieved when necessary. As such, a process instance can be ‘dormant’ for a very long time, without any impact (resource-wise) on running processes.

This approach has two major benefits:

• It keeps the memory footprint of Activiti low. Nothing which is directly needed is offloaded to the database and only fetched when it is absolutely necessary.
• It allows to run Activiti easily in a clustered setup (with a shared database, of course), as no node in the cluster will have any state stored which could be needed by another node. The database is the one and only ‘authentic source’ and when a node in a cluster needs some process state, it is fetched from the database, used, and written back.

There is an exception to that rule: during the execution of an operation of an Activiti service, the MyBatis frameworks (the underlying ORM framework) keeps a ‘session cache’ to avoid fetching the same data twice during the duration of a transaction. However, this cache is very short-lived and does not endanger the ability to run Activiti on a cluster.

Static data

As said, Activiti operations in a stateless way. But there is of course data that will never change, which make it a prime candidate for caching.

A process definition is an example of such ‘static data’. You provide the Activiti engine a BPMN 2.0 XML, the engine parses it to something it can execute (more specifically: a tree of pojo’s corresponding with the structure of the process definition) and stores the xml and some data such as the description, business key, etc in the database (It’s also very important to understand that Activiti does not store the parsed version of the process definition in the database). Such a process definition will never change. Once it’s in the database, the stored data will remain the same until the process definition is deleted.

On top of that, parsing a BPMN 2.0 xml to something executable is quite a costly operation (xml parsing always is). This is why the Activiti engine internally uses a process definition cache to store the parsed version of the BPMN 2.0 xml. As shown on the picture below, the mechanism is pretty easy: whenever the process definition is needed, eg. to start a process instance or to complete a task, the cache is checked and the process definition is only loaded and parsed when needed.

Tweaking the process definition cache

The process definition cache has been part of the first release of Activiti, and hasn’t known any major changes since then. Unfortunately, the cache wasn’t very sophisticated: just a plain dumb hashmap of all process definitions the engine ever touched during the whole uptime of the process engine. For most cases, this isn’t really a problem. You would need an awful lot of process definitions before this becomes a bottleneck.

However, my developer heart bled whenever I passed this class. And last week, I finally decided to spend some time in tweaking the process definition cache. You can find the git commit here: 

Basically, it changes two things:

•  It allows to set a limit to the process definition cache. If such a limit is set, the default hashmap of before will be swapped with an LRU cache with the provided hard limit.
•  It makes the process definition cache (and also the rules cache) pluggable. I’ll explain this bit in a later blogpost.

Limiting the process definition cache

To demonstrate the use of the process definition cache limit, I knocked together a small command line example which you can find on github:

Note that the project already contains some code I will use in my next blogpost. Try to act surprised when you see it then  To build the demo jar, run mvn clean package shade:shade. Go to the target folder and run the demo:

java -jar activiti-procdefcache-demo.jar default

This will boot an in-memory database, boot the process engine and create all the Activiti tables. The process engine will have a cache limit for the process definition cache in the activiti.cfg.xml configuration file:

<bean id="processEngineConfiguration" class="org.activiti.engine.impl.cfg.StandaloneProcessEngineConfiguration">

  ...

  <property name="processDefinitionCacheLimit" value="10" />

</bean>

The application will now ask you how many process definitions you want do deploy. You can fill in any number here. All the process definitions will be the same, only the process definition id will be changed such that a new cache entry will be needed for every such process definition:

After all the process definition have been deployed, the application will now ask you how many process instances you want to start:

Again fill in any number. Don’t expect anything fancy, you’ll just see a lot of logging passing by (the log level is set very low, because the process definition cache logs on a low level and otherwise you won’t see it). But in those loggings, you will often see that the cache limit is hit. The reason for this is that I set the limit pretty low (10, see configuration above), for demo purposes:

And that’s it! That’s all there is too it to set the cache limit, use an LRU cache and tweak the amount of process definition cached in your Activiti engine.

I want to try it!

This feature will be part of the Activiti 5.12 release. If you want to try it already today, you build a snapshot version of 5.12 until we release the real deal. Clone Activiti from github and run the ‘mvn clean install’:

Sidenote: do not use a snapshot version of Activiti on a production database. Chances are high your database will be screwed when upgrading to the actual non-snapshot version.

We’ve released version 5.11 of Activiti just now. It’s a big release, containing many improvements and bug fixes.

Read all about it on Tijs his blog.

Two weeks ago, I gave a talk at the Alfresco Devcon both in Berlin and San Jose. My talk was called ‘Awesome Fast Workflow’, and it definitely was not a  way of me trying to use a catchy title to get more people in the room (honestly! ). I truly am convinced of the work I showed and which we are currently implementing is enabling even more awesome and even faster workflows in Alfresco.

As you all are probably aware of, Activiti is the workflow engine in Alfresco. If it remotely smells like workflow, you can be sure it is handled by Activiti. Since Activiti is ‘just a jar’, it is of course integrated in a way specific to Alfresco. Without going into much technical details, if you want to add a workflow to Alfresco today, you need

• A content model for your tasks that describe the properties, constraints, etc
• Form configs for Share that describe how the forms for your tasks look like
• A BPMN 2.0 process model describing the steps of your process

But weep no more. People that follow my blog for a while now will know that I’ve dabbled with UI-only workflow definitions in the past. And with Alfresco, we’ve got ourselves the perfect target audience for such a tool. A tool that allows to design, deploy and update workflows completely graphically, in a matter of minutes. No technical knowledge required: no XML, BPMN, Java or anything technical. The tool will generate all the xml for you, and put it in the right places. No more chance of having a typo make you lose minutes of your valuable time while the Alfresco server is rebooting.

But – and this is very, very important: the end-results of using such a tool should be usable by developers and technical people. They should be able to take it and run with it the way they do it nowadays.

But enough blahblah. Let me show exactly what I mean with a short screencast that gives you a short version of the demo I gave at the Alfresco Devcons. The context of the demo is the implementation of a simple workflow in a hospital, as shown below.

The point of the demo is not to use a realistic business process, but rather to show how quick and easy it is to create (and afterwards update) workflows in Alfresco. If you’ve implemented workflows in Alfresco before, I’m sure you will appreciate what we’re showing here. I’m not exaggerating if I promised you ‘from design to execution in a matter of minutes’.

See that little ‘cogwheel’ icon in the bottom right? Please press it, and set it to 720p. The quality of the video is much better then!

Important: in the demo I’m using an iPad client. This is just some prototypeware thing. The real meat here is the exposed API. As a first phase, we’ll implement a similar client in Alfresco Share first, before we even start thinking about some mobile client (if any). The real reason is that is dislike (and suck) in html and javascript. And the cool guys do it mobile, anyway .

The source code for the iPad app can be found on https://github.com/jbarrez/Activiti-KickStart. The server-side bit to which this app talks to is a Java maven application, found at https://github.com/jbarrez/activiti-kickstart-ios. You’ll need a ‘modern’ Alfresco community to make it work. Anything 4.2.a+ will do.