My Five Rules for Remote Working

A couple of weeks ago, there was a stir (again) about remote working and its succes and/or failure: it was reported that Reddit, the website where many people lose countless of hours, were forcing all their employees to move to SF. After a similar thing happened at Yahoo last year it made me think about why remote work is such a huge success for us at Activiti and Alfresco.

You see, I’m a remote worker for more than five years now. First at Red Hat and then at Alfresco. I worked a couple of years as Java consultant before that, so I’ve seen my share of office environments (checking my Linkedin, it comes down to about 10 different office environments). I had to go to these offices each day.

Comparing those experiences, I can – without exaggeration – say that I’m way more productive nowadays, working from home. Many people (both in and outside IT) ask me how I do it. They say “they couldn’t do it”. Maybe that’s true. Maybe some people need a lot of people around them. But for the kind of job I am into – developing software – I believe having a lot of people around me doesn’t aid me in writing higher quality software faster.

Anyway, like I said, I did some thinking around it and I came to the following “rules” which I have been following all these years which I believe are crucial (at least for me!) to making remote working a success.

dilbert_remote

(comic from http://www.dilbert.com/ )

Rule 1: The Door

Having a separate space to work is crucial when wanting do serious remote working. Mentally it is important that you can close “The Door” of your office space when you finished working. It brings some kind of closure to the working day.

Many people, when they work from home, put their laptop on let’s say the kitchen table. That doesn’t work. It is not a space that encourages work. There are distractions everywhere (kids that come home, food very close by, …). But most importantly, there is no distinction between when you are working and when you are not.

My wife and kids they know and understand that when The Door is closed, I’m at work. I can’t be disturbed until that Door opens. But when I close The Door in the evening and come downstairs, they also know that I’m fully available for them.

My door!

My door!

Rule 2: The Gear

The second rule is related to the first one: what to put in that room. The answer is simple: only the best. A huge desk, a big-ass 27″ monitor (or bigger), a comfortable chair (your ass spends a lot of time on it), the fastest internet you can buy, some quality speakers, a couple of cool posters and family pictures on the wall, ….

This is the room where you spend most of your time in the week, so you need to make it a place where you love to go to.

My setup

My setup

Often, I hear from people which company allows for remote work that their company should pay for all of this. I think that’s wrong. It’s a two-way street: your company gives you the choice, privilege and trust to work from home, so you from your side must take care that your home office isn’t decreasing anything compared to the office gear you have. Internet connection, chair and computer monitor are probably the most important bits here. If you try to be cheap on any of those, you’ll repay it in decreased productivity.

Rule 3: The Partner

Your partner is of utmost importance to make remote work a success. Don’t be fooled by the third place here, when your partner is not into it, all the other points are useless.

It’s pretty simple and comes down to one core agreement you need to make when working from home: when you are working from home you are not “at home”. When you work, there is no time for cleaning the house, doing the dishes, mowing the grass, etc … You are at work, and that needs to be seen as a full-time, serious thing. Your partner needs to understand that when you would do any of these things, it would be bad for your career.

Many people think this is easy, but I’ve seen many fail. A lot of people still see working from home as something that is not the same as “regular work”. They think you’ve got all the time in the world now. Wrong. Talk it through with your partner. If he/she doesn’t see it (or is jealous), don’t do it.

Rule 4: Communicate, communicate, communicate

More than a team in an office, you need to communicate. If you don’t communicate, you simply don’t exist.

At Activiti, we are skyping a lot during the day. We all know exactly what the other team members are currently doing. We have an informal agreement that we don’t announce a call typically. You just press the ‘call’ button and the other side has to pick it up and respond. It’s the only way remote work can work. Communicate often.

Also important: when you are away from your laptop, say it in a common chat window. There is nothing as damaging for remote workers as not picking up Skype/Phone for no reason.

Rule 5: Trust People

The last rule is crucial. Working remote is based on trust. Unlike in the office, there is no physical proof that you are actually working (although being physically in an office is not correlated with being productive!). You need to trust people that they do their job. But at the same time, don’t be afraid to check up on people’s work (for us, those are the commits) and ask the questions why something is taking longer than expected. Trust grows both ways.

The second part of this trust-story is that there needs to be trust from the company to the team. If that trust is missing, your team won’t be working remote for long. At Activiti, we are very lucky to have Paul Holmes Higgin as our manager. He is often in the office of Alfresco and makes sure that whatever we are doing is known to the company and vice versa. He attends many of the (online) meetings that happen company wide all the time so that we are free to code. There is nothing as bad for a remote team as working in isolation.

trust_fall

Conclusion

So those are my five (personal!) rules I follow when working from home. With all these bad press from the likes of Reddit and Yahoo, I thought it was time for some positive feedback. Remote work is perfect for me: it allows me to be very productive, while still being able to see my family a lot. Even though I put in a lot of hours every week, I’m still seeing my kids grow up every single day and I am there for them when they need me. And that is something priceless.

Does this sound cool to you? Well, at Alfresco we are still hiring people to work on Activiti!

Webinar ‘Process Driven Spring Applications with Activiti’ now on Youtube

As I mentioned, I did a webinar on Spring Boot + Activiti last week (at 6 am …. yes, it hurt) with my good pal Josh Long.

If you missed it, or want to see the awesomeness again, here’s the recording:

 

On a similar note, the webinar that Josh did before this one, on Spring Boot / Spring Cloud is *really* awesome. All this micro-service architecture stuff suddenly made a lot of sense and all the pieces of the puzzle fell together by watching this webinar. Go check it out (after the Activiti webinar, obviously ;-)), it’s well worth your time.

Upcoming Webinar: Process Driven Spring Applications with Activiti – Sept 23rd

oss-logo-spring

Next week, I’ll be doing a webinar together with my friend Josh Long (he’s a Spring Developer Advocate, committer to many open source projects and of course Activiti). I will show some of the new Activiti tooling we’ve been working on recently, while Josh will demonstrate with live coding how easy it is to use Activiti in Spring Boot (spoiler: really easy).

You can register for the webinar for free here: https://spring.io/blog/2014/07/29/webinar-process-driven-spring-applications-with-activiti-sept-23rd

One day later, the Alfresco Summit will be kicked off in San Francisco. I’m joining two talks there:

For those who can’t make it to San Francisco: don’t worry, we’ll be landing in London two weeks later, Oct 7-9!

Seriously reducing memory consumption when running multiple Activiti Engines on the same JVM

The Activiti Forum is a place where I typically try to spend every day a bit of time. Surely, it is hard work to keep up with the influx of posts (but then again it means that Activiti is popular, so I’m not complaining!) but sometimes you find a true gem amongst the posts.

On the Forum, user jkronegg (or Julien, as he signs his posts) has posted an interesting finding which I’d like to give some more attention here. His use case is running multiple Activiti Process Engines on the same JVM. Maybe for multi-tenancy (one engine for each customer?), but that’s not the point here.

As you can read in his post, he has a setup of 50 Activiti Process Engines in the same JVM. And, while Activiti uses very little memory, he was interested if it could be brought down. More specifically, after profiling, he found that for every Activiti Process Engine most of the memory was consumed by the Mybatis internal Configuration object (Activiti uses Mybatis as database access framework). Conceptually, this configuration objects holds a parsed version of the Mybatis mapping files. As jkronegg correctly analyses, this data is actually static: two process engines configured the same will basically duplicate this Mybatis sql mapping information.

So, to verify this claim, I wrote a simple console application that you can find on Github. This little application gives you 3 options: start a number of ‘regular’ Activiti process engines, start a number of engines with a ‘shared mybatis config’ (I’ll come back to this) and a last option to check if you can actually use the engines (ie. a process deploys and runs).

01

So, time to boot up some ‘regular’ Activiti Engines, let’s say we boot up 250 regular Activiti Engines. I also have a profiler running (simply VisualVM, shipped with every JDK) to see the memory consumption:

02

You can clearly see that the memory nicely gets filled to about 1.5-1.6 GB (still only about 6MB for each engine).

When we calculate the objects with the biggest retained sizes, we can see that jkronegg is correct: the Configuration instances of Mybatis do take up 98.9% of all used memory!

03

Now, let’s see what happens if we apply the changes described in the forum post. More specifically, there are two main culprits that account for most of the memory usage (see the post for more details). This is proven when we inspect one of the instances: two objects (sqlFragments and mappedStatements) take up most of the memory of a Configuration instance:

04

By setting the reference on a new Mybatis Configuration object using reflection, the static data effectively gets shared between all next engines that get booted up. I did make a small change to the Activiti engine to make this easier in the future (with the current version you need to copy and paste the whole block of initSqlSessionFactory).

See SharedMyBatisProcessEngineConfiguration on Github for the implementation of the idea of copying the reference of those heavy objects (pretty much the same as described in the forum post).

If we now do choose option 2 in the console application, and boot up again 250 Activiti Engines, the memory  consumption looks as follows:

05

We’re not even touching the 250MB line (which would be 1MB / engine!) Only a fraction of the memory is being used, while still having exactly the same functionality!

and if we look at the number of instances and its retained size, we do indeed see there are 250 instances of the org.apache.ibatis.session.Configuration class, but now only account for 15,1% (was 98.9% before!)

06

Conclusion

So what can we learn about all of the above? Well, first of all, the reason why I posted this on my blog is because I simply love deep technical stuff like this. I’m not saying you should use this in production. You can be the judge of that yourself, it’s fiddling with reflection, depending on Mybatis internals and not tested beyond this simple example :-). But it is damn cool that you can quickly boot up so many Activiti engines with so little memory!

But what this really shows is the power of open source software. Both Activiti and Mybatis are open source and combine that with intelligent and creative developers you can get a whole lot further than with anything closed. Secondly this also demonstrates the awesomeness of the Activiti open source Community. There are a lot of people in our community, and by exchanging ideas and findings like above, Activiti gets better every day.

Thank you for that.

Running Activiti on JDK8

… just works!

All the Activiti code is written with JDK6 compatibility in mind, so it’s no surprise it just works out of the box.

Scripting

There are however some subtle changes when you use Script Task with javascript. In JDK 8, the Rhino engine has been replaced with Nashorn (which is a very good thing!).

nashorn

 

Our QA did show us that some of our unit tests with javascript script task did not pass on JDK 8. But don’t worry, the changes are minimal. More precise, we found two things

1. importing packages

In JDK 6/7 you can import a java package and then use classes from that package in your javascript (eg to use an arraylist):

importPackage (java.util);
var myVar = new ArrayList();

This doesn’t work on Nashorn and thus JDK 8. However, the nice developers of Nashorn have provided a way to support this in a backwards compatible way by loading a certain file:

load("nashorn:mozilla_compat.js");
importPackage (java.util);
var myVar = new ArrayList();

And everything will work again nicely.

2. Implicit variable fetching

This one is more subtle. On JDK 6/7 you could do this

var outcome = task.getVariableLocal('outcome');
if (outcome == 'approve') {
  task.setCategory('approved');
} else {
  task.setCategory('rejected');
}

However, due to an error in our unit test, we didn’t actually set the variable on local scope but on process instance scope. Still, the test worked on JDK 6/7 and failed on JDK8!

It took me some deep debugging, but finally I found the error. In JDK6/7, the getVariableLocal(‘outcome’) does resolve to null too. However, it does a fetch of getVariable(‘outcome’) immediately afterwards. This does not happen on JDK8.

There is a subtle thing going on here: on JDK6/7 something like

var outcome;

or

var outcome = null;

triggers a lookup in the ScriptingBindings, which defaults to a getVariable() call in Activiti. The script bindings call does not happen on JDK 8, which is actually more correct! So this is actually something to be aware of when using JDK6/7.

TL;DR

Activiti just works on JDK 8. If you are using javascript in your script task, be aware of subtle changes.



Review ‘Activiti 5.x Business Process Management’ by Zakir Laliwala and Irshad Mansuri

I’ve been contacted by the people of Packt Publishing to review their recent book release of the ‘Activiti 5.x Business Process Management”, written by Dr. Zakir Laliwala and Irshad Mansuri. For an open source project, books are a good thing. They indicate that a project is popular, and often people prefer books over gathering all the information from bits all over the internet. Don’t expect any insight into use cases, process modeling best practices or how to fit Activiti architecturally though, this is a hands-on description on how you get started technically with Activiti.

The book, as is clear from the cover, clearly aims to people who want to get started with Activiti and have no prior knowledge of it:

7065OS_Activiti 5.x Business Process Management Beginner's Guide

 

So, before going into more details, I’m happy to report that the book does exactly that. It introduces Activiti and its concepts gently and with a lot of screenshots of installing components but also when executing certain code. For people who are well advanced in Activiti certain things are a given, but I can imagine for people that have to start from zero these steps help to get bootstrapped quicker and with a lot less pain during learning.

In the past, I’ve read (and reviewed) books from Packt where the English wasn’t good to a point where it hindered reading it. However, the English in this book is really well-written and understandable. Absolutely no complaints here.

Some of the remarks I noted down some small remarks while reading the book:

  • The XML formatting of the process XML is often not readable due to lack of indentation and formatting. Probably a problem in general for printed books, but it’s very hard to understand the processes without structure.
  • I liked the fact that many concepts in the book were accompanied with a unit test, something I definitely support!
  • The bit about reporting had a bad screenshot, which kinda defeats the purpose of showing reporting
  • In chapter 5, it seems to hint that the activiti.cfg.xml way of configuring the engine is ‘Spring-only’, but actually this can be used outside of spring (only relying on spring-beans to have the parsing)
  • Chapter 6 about the services was really good: clearly explained and always with a short example/unit test. Also explains how to do debugging of processes. Many people will find this chapter useful
  • Chapter 7 was the same, but now for REST. Decent explanation of rest client install *and* unit testing with an http client. Nice.
  • Chapter 8 (integration with Liferay/Drools/OSGI) felt a bit ‘off’. It’s something I would have placed at the end of the book, and I definitely would have placed chapter 9 (advanced Activiti constructs) before as it fits better. Also, it doesn’t really go deep, it’s more about how Activiti is integrated with those technologies, accompanied with screenshots of the installation process. No insight into use cases or how it would be used in reality (granted, which is probably not the goal of this book).

So in summary: I cheer for everything being written about Activiti, so I really like the fact that this book was written. But more-so, I think it targets the beginners audience well. It has one goal: get a technical someone going with Activiti quickly and teaching the basics. I believe  people who are in the dark about Activiti and need to get started will be able to get going quickly with this book.

Important: Activiti 5.15 and MySQL 5.6+ users

Execute Custom queries in Activiti

(This will probably end up in the user guide of the Activiti 5.15 release, but I wanted to share it already)

The Activiti API allows for interacting with the database using a high level API. For example, for retrieving data the Query API and the Native Query API are powerful in its usage. However, for some use cases they might not be flexible enough. The following section described how a completely custom SQL statement (select, insert, updated and deletes are possible) can be executed against the Activiti data store, but completely within the configured Process Engine (and thus levering the transaction setup for example).

To define custom SQL statements, the Activiti engine leverages the capabilities of its underlying framework, MyBatis. The first thing to do when using custom SQL, is to create a MyBatis mapper class. More info can be read in the MyBatis user guide. For example, suppose that for some use case not the whole task data is needed, but only a small subset of it. A Mapper that could do this, looks as follows:

public interface MyTestMapper {

  @Select("SELECT ID_ as id, NAME_ as name, CREATE_TIME_ as createTime FROM ACT_RU_TASK")
  List<Map<String, Object>> selectTasks();

}

This mapper must be provided to the Process Engine configuration as follows:

...
<property name="customMybatisMappers">
  <set>
    <value>org.activiti.standalone.cfg.MyTestMapper</value>
  </set>
</property>
...

Notice that this is an interface. The underlying MyBatis framework will make an instance of it that can be used at runtime. Also notice that the return value of the method is not typed, but a list of maps (which corresponds to the list of rows with column values). Typing is possible with the MyBatis mappers if wanted.

To execute the query above, the managementService.executeCustomSql method must be used. This method takes in a CustomSqlExecution instance. This is a wrapper that hides the internal bits of the engine otherwise needed to make it work.

Unfortunately, Java generics make it a bit less readable than it could have been. The two generic types below are the mapper class and the return type class. However, the actual logic is simply to call the mapper method and return its results (if applicable).

CustomSqlExecution<MyTestMapper, List<Map<String, Object>>> customSqlExecution =
    new AbstractCustomSqlExecution<MyTestMapper, List<Map<String, Object>>>(MyTestMapper.class) {

  public List<Map<String, Object>> execute(MyTestMapper customMapper) {
    return customMapper.selectTasks();
  }

};

List<Map<String, Object>> results = managementService.executeCustomSql(customSqlExecution);

The Map entries in the list above will only contain id, name and create time in this case and not the full task object.

Any SQL is possible when using the approach above. Another more complex example:

  @Select({
    "SELECT task.ID_ as taskId, variable.LONG_ as variableValue FROM ACT_RU_VARIABLE variable",
    "inner join ACT_RU_TASK task on variable.TASK_ID_ = task.ID_",
    "where variable.NAME_ = #{variableName}"
  })
  List<Map<String, Object>> selectTaskWithSpecificVariable(String variableName);

Using this method, the task table will be joined with the variables table. Only where the variable has a certain name is retained, and the task id and the corresponding numerical value is returned.

This will be possible in Activiti 5.15. However, the code (and more specifically the Command implementation and the wrapper interface) can be used in any older version of Activiti.

Brazos For Activiti

As I mentioned a while ago, BP3 has become on of our enterprise partners for Activiti. The people of BP3 always said that they wanted to contribute to the open-source code of Activiti, but of course the proof of the pudding is in the eating.

We’re very enthousiast about what they’ve shown us recently: a full fledged drag and drop UI builder for Activiti forms, based on their experiences with building such a product on top IBM BPM (but now of course way cooler). I wanted to give it the exposure it deserves: see it for yourself!

And quoting Scott Francis,

we’re going to release this into the open source community on the same licensing terms as Activiti (Apache).  We think this could really elevate the way people approach building process-enabled application UIs with Activiti, and it seems like a great first contribution from BP3 to the Activiti open source community.

That’s just awesome. Stay tuned!

Advanced scripting in Activiti: Custom Configuration Injection

The scripting task is probably one of the ‘oldest’ classes in the Activiti code base, but I think it is still underused by many. The (perceived?) downsides are of course performance (interpretation vs compilation) and less support from IDE perspective.

However, the benefits (imo) outweigh this:

  • Scripts are defined in the process xml itself. No more worries about versioning and having to juggle with libs on the classpath.
  • What we’ve seen in the past is that less technical skilled people dare to try scripts. But never Java.

Anyway, what few people know or have realized is that you can do really awesome and advance stuff in scripts in Activiti. Since such a script is executed within the process engine, you have access to everything the engine is capable of. Yes … everything… which makes it both a very powerful but also (potential) dangerous thing (if you don’t know what you’re doing).

Let me walk you through such an example. I like to call it ‘custom configuration injection’ as a concept, because it effectively allows you to add custom logic at runtime which alters process execution significantly. If you have a cooler name for it, please let me know.

All code can be found on my Github Page: https://github.com/jbarrez/activiti-advanced-scripting

awesome-code-648x303

The use case

Now what is this thing I want to do. Well, I want to have a process that, when executed

  • Adds a ‘task completion event handler’ to every user task that is executed
  • This event handler must fire a custom event off to a remote URL, where potentially a event processor is doing its stuff

So basically, we want to fire off custom events to some remote URL whenever a task gets completed. A good use case for this could be Business Intelligence reporting/Complex event processing, eg with something like Esper.

Screen Shot 2013-07-23 at 10.03.11

The first version

The first cut of this functionality can be found at https://github.com/jbarrez/activit-advanced-scripting/blob/master/src/test/resources/org/activiti/test/my-process.bpmn20.xml. When this process is executed, the following happens:

var config = Context.getProcessEngineConfiguration();
var bpmnParser = config.getBpmnParser();

We simply fetch the current ProcessEngineConfiguration instance. We fetch the BpmnParser instance from this configuration, as we will want to change the general user task parsing for the whole engine.

Next, we build the script:

var script = "";
script = script + "importPackage(java.net);";
script = script + "importPackage(java.io);";
script = script + "var url = new URL('http://localhost:8182/echo');";
script = script + "var connection = url.openConnection();";
script = script + "connection.setRequestMethod('POST');";
script = script + "connection.setDoOutput(true);";
script = script + "var outputStream = new BufferedOutputStream(connection.getOutputStream());";
script = script + "outputStream.write(new java.lang.String(\"{'eventType':'task-complete'}\").bytes);";
script = script + "outputStream.flush();";
script = script + "connection.connect();";
script = script + "var respCode = connection.getResponseCode();";
script = script + "if (respCode != 200) ";
script = script + "println('Response code : ' + respCode);";
script = script + "outputStream.close();";
script = script + "connection.disconnect();";

This is obviously not the most efficient way to do this, but it sure shows the details of what happens. The message ‘eventType:task-complete’ is send to the localhost:8182 url through standard java.net and java.io classes.

The tricky part comes next:

var handler = new ExecuteScriptOnTaskCompleteBpmnParseHandler("JavaScript");
handler.setUserTaskCompleteScript(script);
bpmnParser.getBpmnParserHandlers().addHandler(handler);

// reset the deployment cache such that the new listener gets picked up on a new redeploy
config.getProcessDefinitionCache().clear();

Here we add a BpmnParseHandler class to the engine configuration. The parse handler will add the execution of the script defined above to every receival of the ‘task complete event’ send out by the engine. This parse handler kicks in every time a user task is parsed, which effectively adds our ‘send-event-to-remote-service’ to every user task now happening in your Activiti environment!

There is a unit test to see how this works: https://github.com/jbarrez/activiti-advanced-scripting/blob/master/src/test/java/org/activiti/test/ExecuteScriptInProcessTest.java. In the test, we setup a very simple ‘echo service’ which simply prints out whenever such an event is received. If you run it in your IDE, you’ll see something like this:

Screen Shot 2013-07-23 at 09.53.00

 

But we can do better

But we can do better. Check the following code.

var handler = new ExecuteScriptOnTaskCompleteBpmnParseHandler("JavaScript");
handler.setUserTaskCompleteScript("http://localhost:8182/scripts/task-complete.js");
handler.setExecuteScriptInJob(true);
bpmnParser.getBpmnParserHandlers().addHandler(handler);

// Update the configuration to use the correct job handler
var jobHandler = new ExecuteScriptJobHandler();
config.getJobHandlers().put(jobHandler.type,jobHandler);

This code does the same as in the previous section, ie. attaching a listener for ‘complete’ events to every user task. However, this implementation:

  • Executes the script asynchronously
  • Does not define the script in the process xml, but it is fetched from a remote url
  • Updates the job handler configuration

If you ask me, that’s pretty awesome! So this means that the actual sending of a message to the remote service is not impacting the execution performance of your process instance. Obviously, from here you can go crazy and add persistent queues and all that fanciness. And on top of that, the script is always fetched from a remote server. If you want to update the logic that is executed, simply change the script that is returned. This means you can impact process execution AT RUNTIME without touching the actual process.

There is a unit test for this at https://github.com/jbarrez/activiti-advanced-scripting/blob/master/src/test/java/org/activiti/test/ExecuteScriptWithJobTest.java

If you run this test,  you’ll see the following. Note that we host the completion script as static file called ‘task-complete.js’ on the test server.

Screen Shot 2013-07-23 at 09.50.36

In the test, you can see we have to execute the async job specifically to see the output of the test.

Caveat

On small caveat here: when the process engine reboots, the configuration will be reloaded from config file. Hence, the process from above that injects custom logic is not added. However, this can easily be done by using a ProcessEngineLifeCycleListener implementation that executes a process definition of a certain category after the engine has booted up. If you for example give all these processes ‘config-processes’ as category, they can easily be executed on bootup.

Conclusion

Scripting in BPMN 2.0 processes is a very powerful feature. It allows you to change process execution engine-wide in a matter of a few lines. Of course, all the code above can be done with Java. But the examples above use nothing more than standard BPMN 2.0 and the javascript engine that is bundled with every JDK install.

Thanks for reading. Happy coding!