Earlier this week, Amazon announced that Amazon Aurora is generally available on Amazon RDS. The Aurora website promises a lot:
Amazon Aurora is a MySQL-compatible, relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. Amazon Aurora provides up to five times better performance than MySQL at a price point one tenth that of a commercial database while delivering similar performance and availability.
Here at Alfresco, my colleagues have been playing around with Aurora too – actually Alfresco is mentioned as a partner on the announcement article above, so ‘playing’ is probably not the right word ;-). But anyway, the noise I’ve heard coming from that team has been quite positive. Since we’ve just published the results of running our performance benchmark on (amongst others) Oracle on Amazon RDS, trying out the benchmark on Aurora was a no-brainer.
I’ve added the benchmark results to the sheet of the previous performance blog post: https://docs.google.com/spreadsheets/d/15wK7dQfgChz-9XF1qYi0qMnNoRWp4FQQr6XTYHXTHEE/edit#gid=739564290. The sheet compares the numbers from Oracle on RDS vs Aurora on RDS.
We selected the same hardware as the benchmark on Oracle RDS. Remember from my previous post that I couldn’t get MySQL to perform as expected on RDS (hence why I switched to Oracle on RDS, which surely is pricier!). Luckily, Aurora doesn’t seem to suffer the same fate as it’s MySQL nephew
So here’s what we saw: when using just a single thread, the numbers weren’t that clear. Half of the use cases performed better on Oracle, the other half better on Aurora. On average, Aurora was 16.96% faster for the same benchmark on Oracle. However, our test with the most data (service tasks vars 02), which is hard to see on the chart due to the scale, performed 50% better.
When we add more threads to the system to execute processes, we see a clear pattern arise. The more threads we added, the better the results for Aurora became. With ten threads (the most we tested), Aurora is faster in all but two cases (which does makes me want to check upon those tests, cause they are measured in a different way), and in general 25% faster.
Lastly, we tested the random execution (2500 random process instance executions), which comes close to a real system with real processes running on it. The graph clearly shows that Aurora beats Oracle for this test. On average, we saw a 20% better performance of Aurora:
Although we didn’t got the five times performance improvement from the marketing announcement (EDIT 31st July 2015: as pointed out in the comments below, the ‘up to five times’ is for MySQL and NOT for commercial databases. Which matches what we saw.), I was genuinely surprised that, with the same hardware and setup, Aurora is 20% faster on average with outliers up to 81%. Also note that our benchmark is really write-intensive, where Aurora makes setting up read replica’s a breeze through the AWS console. So probably for other benchmarks, with a bit more reads, Aurora can shine even more.
Also cost-wise, Aurora is interesting. I checked the bill: Oracle was about 1.96$/hour while the Aurora machine I was using is advertised at 1.28$.hour (couldn’t verify, as it shows as 0$ on the bill currently … maybe a promotion?). So more performance for less money … who doesn’t like that!
When it comes to Activiti, the conclusion is still the same: Activiti is highly performant and scalable. A faster database just makes that event better
It’s been three years since we published the ‘The Activiti Performance Showdown‘, in which the performance of the Activiti engine (version 5.9 and 5.10) was benchmarked. Looking at my site analytics, it’s high in the top 10 of most read articles: up until today, 10K unique visits with an average time of 7 minutes (which is extraordinary in this day and age!).
After three years it is time (or perhaps already long overdue) for updating the benchmark results using modern hardware with the latest Activiti engine. We’ve obviously been enhancing and improving the engine with every release, but in the latest release something called ‘bulk insert‘ was added to the core engine.
We worked together with our friends at Google for implementing this feature (they actually did the bulk of the work, no pun intended) so a big shout-out and thanks to Abhishek Mittal, Imran Naqvi, Logan Chadderdon, Bo-Kyung Choi and colleagues for our past discussions and code! As you can imagine, the scale at which Google uses Activiti makes for interesting topics and challenges :-). In fact, some major changes in the past (for example the new Async job executor) and very interesting items on the future roadmap were driven by discussions with them.
Edit (July 30th 2015): ran the benchmark on Amazon Aurora. Results here: http://www.jorambarrez.be/blog/2015/07/30/activiti-performance-showdown-aurora/
The code used to gather the numbers are the same as three years ago and on Github: https://github.com/jbarrez/activiti-benchmark
To run the benchmark, simply follow instructions on the Github page. Basically what the test does:
We let the benchmark project mentioned above loose on the following setups:
Setup 1 and 2 are interesting cause they show what is the raw overhead of the engine, without any network delays (as its on the same machine). Of course numbers will vary with different databases, but the main point to illustrate with this benchmark is to demonstrate how light Activiti is.
Setup 3 is in my eyes probably even more interesting (if that’s even possible), as it runs on ‘real’ servers with real network delay. In our experience, AWS servers aren’t as performant as real physical servers, so that’s only good news when you look at the numbers and are already impressed ;-).
Furthermore, all tests were ran in Spring mode (the engine used in a Spring setup) and with a BoneCP connection pool (I used BoneCP, although I know it has been superceded by HikariCP, but wanted to have the same setup as three years ago).
When looking at the results, they are obviously faster than the similar 2012 numbers for the same processes. Obviously the engine has improved and hardware vendors didn’t sit still either. Generally, I noticed a 2x – 4x improvement in the throughput results.
The tests were also ran on Activiti 5.17 and 5.18. So we learn two things:
I bundled all benchmark results in the following Google spreadsheet:
If you prefer Excel:
It has various sheets which you can select below: a sheet for the different test environments and then sheets for each of the tested processes in detail (with pretty graphs!). Each of the sheets contain analysis text at the top. Anyway, let’s look at the different sections in a bit of detail.
Note that the numbers here only show the throughput/second (aka process instances started and completed per second). Way more numbers (including the certainly interesting average timings of process instance execution) can be found in the following links:
I’ve split the results up in two types: the first sections will cover the difference between 5.17 and 5.18 and will highlight the difference the bulk insert makes. In the next sections, we’ll look at the numbers for the process definitions in detail for 5.18.
This maps to the first sheet in the spreadsheet linked above.
When looking at the numbers, we learn that Activiti adds very little overhead when executing processes. There are crazy numbers in there like +6000 process instances / second (for very simple process definition of course). Keep in mind that this is with a local MySQL. But still, this proves that the overhead of the Activiti engine is very minimal.
The sheet does show a bit of red (meaning 5.18 has a lower throughput) here and there. This is logical: there is a minimum of data being produced (no history) and the average timings are sometimes really low (a few milliseconds in some cases!). This means that doing a bit of extra housekeeping in a hashmap (which we do for bulk insert) can already impact the numbers. The ‘red’ numbers are in general within 10% range, which is ok.
In my view, the ‘random’ execution (2500 randomly chosen process definitions using an equal distribution, throughput/second) is the most interesting one, as in a typical system, multiple process instance of different process definitions will be executed at the same time. The graph below shows that 5.17 and 5.18 are quite close to each other, which was to be expected.
There are however two outliers in the data here: the service task vars 1 and 2 process definitions. These process definitions are pretty simple: they have a 7 steps (version 2 has a user task) and one of them is a service task that generates 4 and 50 process variables respectively. For these two cases, with more data (each process variable is an insert) we can see an increase from 20% up to 50%! So it seems like the bulk inserts starts to play here. That makes us hopeful for the next tests with historic data!
This maps to the second sheet in the spreadsheet linked above.
In the benchmark we did three years ago, we already learned history has a huge impact on performance. Of course, most people want to run with history enabled, as this is one of the main benefits of using a workflow engine like Activiti.
As we suspected from the previous setup, enabling history makes the bulk insert shine. In general we see nice improvements, in the 5%-40% range. Tests with fewer data (eg the user task tests, which do a lot of task service polling) see a bit of red there (meaning lower performance), but this is probably due to the same reasons as mentioned above and are well withing acceptable (<10% range). So this proves that the bulk insert, when there is a ‘normal’ amount of data being generated does have a very good impact on performance. Also, in general we see again that the overhead of the Activiti engine is very minimal with very nice numbers for the throughput/second (think about it. Doing 100’s of process instance executions per second is a lot when you translate this to process instances / day!).
The random execution graph (same as above, 2500 random executions, throughput/second), now shows that 5.18 keeps its distance from 5.17. For all threadpool settings, 5.18 can do more process instance executions in the same time period.
This maps to the third sheet in the spreadsheet linked above.
Now here it gets different from the tests we did three years ago. We have two servers, both in the cloud (Amazon), both with real network delays. One thing to notice here is that the throughput/seconds is lower than the previous setup (about 1/4 of the throughput). Again, real network delays of the Amazon infrastructure are in play here.
The reasoning behind the bulk insert idea was that, when switching to a data center with a higher network delay, it would be beneficial if we can squeeze in more work for each roundtrip we do. And oh boy, that surely is the case. If you look at the numbers in the sheet, they are in general almost green or < 5% difference. But more importantly, we see improvements up to 80% for some cases (not surprisingly those cases with a lot of data being produced)!
The random execution graph (same setup as above, 2500 random executions, throughput/second) shows the following:
On average, there is a 15% gain for this particular test (which is awesome). Also interesting here to see is the shape of the graph: a linear line, diverting from the 5.17 line the more we threads we add. But if you scroll up, you can see the shape being very different when running on my laptop. In fact, you can state that we’ve reached a tipping point around 8 threads for the previous two graphs while here there is no sign of tipping yet. This means that we could probably load the Oracle RDS database way more than we were doing. And that my laptop is not a server machine, which yeah, I already knew ;-).
For each of the process definitions, there is a specific sheet in the spreadsheet linked above.
process01 is the simplest process you can imagine:
process02 is a bit longer, all still passthrough activities, but this generates obviously more history (for example one entry for each passthrough activity):
process03 tests the parallel gateway (fork and join), as it’s a more expensive activity (there is some locking going on):
process04 goes a bit further with the parallel gateways, and tests the impact when using multiple join/forks at one:
process05 is about using a service task (with a Java class) together with an exclusive gateway:
multi-instance is a process definition that tests the embedded subprocess construct, with a parallel gateway in the middle. Knowing a bit about the internals of the Activiti engine, this process most likely will be the slowest:
usertask01 to 03 are quite similar, 01 has just one user task, 02 has seven sequential ones and 03 has two user task, with a parallel gateway as fork join.
And lastly, the servicetask-vars01/02 process definitions have the same structure as usertask02, but with a service task that generates process variables (four process vars for the 01 version, fifty process variables for the 02 version). The 02 version also has a user task.
The first batch of process definitions we’ll look at is process01/02/03/04/05. These are all process definitions that complete after starting a process instance, no wait state. Which makes it a prime candidate for testing the raw overhead the Activiti engine adds.
Here are the throughput and average timing numbers (click to enlarge):
And the similar numbers for the Oracle RDS setup (click to enlarge):
These numbers show two things clearly:
The second batch of process definitions are those with a user task (I’ve made a crude distinction here, as looking more detailed would add way too much information). Now, one big difference with the previous batch is the way the time is measured here. For the previous batch, it was easy, as the process instance would complete after start. For these process definitions however, the process instance reaches a wait state. The numbers below take in account both the starting of the process instance and the task completions. So for usertask02 for example, with seven user tasks, this means that in the results you have to remember that we are talking about 8 database transactions (one for process instance start, seven for each user task). Which explains the difference in magnitude of the numbers versus the previous batch.
The first graphs show the results for 5.18 on a local Mysql (click to enlarge):
Similar, but now for Oracle (click to enlarge):
A similar analysis as with the previous batch can be made: when adding more threads, the average time goes up, but in the same time period a lot more work can be done. Still, taking into account we’re talking here about multiple database transactions for each process instance run, the numbers are really good.
Now, this was only a summary of the number analysis we did. The spreadsheet contains way more data and graphs (but we have to limit this very long blog a bit…).
For example, the spreadsheet shows the detailed analysis for each process definition in detail. Take for example the throughput graph of process definition process02 (seven manual tasks). Above is the local Mysql, below is the Oracle RDS:
First of all, the difference in throughput between 5.17 and 5.18 is very noticeable here (since there is quite a bit of historic data). But also the numbers are really impressive, in both setups, going well over the thousands for the local db and hundreds for Oracle RDS.
Another example is the servicetask-vars02 process definition (seven steps, one is a service task that generates fifty variables, and one is a user task to make sure all variables are written to the database – they can be optimised away without wait state otherwise). Again above is the local Mysql, below is Oracle RDS:
The difference in line is very interesting here. It shows the Oracle is having no issue with the additional concurrent load. It also nicely shows the difference between 5.17 and 5.18 again. Do take in account that in this process fifty process variables get written. While the numbers are a magnitude lower than the previous example, taking in account what is happening makes the numbers look very good in my opinion.
The conclusion which we made three years ago, is still valid today. Obviously results are better due to better hardware, but I hope the graphs above prove it’s also about continuously improving the engine based on real-life deployments. Again, if you missed it in the introduction above, the bulk insert was conceived as a solution for a problem Google found when running Activiti at scale. The community is really what drives Activiti forward by challenging it in every way possible!
In general we can say
Like I said three years ago: the numbers are what they are: just numbers. Results will vary from use case, hardware and setup. My main point which I want to conclude here, is that the Activiti engine is extremely lightweight. The overhead of using Activiti for automating your business processes is small. In general, if you need to automate your business processes or workflows, you want top-notch integration with any Java system and you like all of that fast and scalable … look no further!
As promised in my previous post, here are the recordings of the talks done by our awesome Community people.
The order below is the order at which they were planned in the agenda (no favouritism!). Sadly, the camera battery died during the recording of the talks before lunch. As such, the talks of Yannick Spillemaeckers (University of Ghent) and Elmar Weber (Cupenya) were not usable :-(. Their slides can be found online though:
Here are the recordings of the talks. The slides can be found here.
The Activiti Community Day in Paris last week was a blast. The venue, the talks, the attendees … all were top notch.
For the people that weren’t able to attend this time (I’m sure you had good excuses … right?), don’t worry. We’ve recorded most of the sessions (and the slides are already online a couple of days). The first batch of recordings, those about Activiti 6, are ready and on Youtube.
The rest of the sessions will follow in the next couple of days.
This presentations goes into what Activiti v6 is, what the changes are and why we slap a 6 sticker on it.
V6 will be shipped with a new UI. This presentation shows it live in action, running against the v6 engine.
The questions after our presentations. You can clearly see how much more relaxed we are after the two presentations beforehand all went very well :-).
The Activiti engine was born five years ago and we started with version 5.0 (a cheeky reference to our jBPM past). In those five years we’ve seen Activiti grow beyond our wildest dreams. It is used all across the globe in all kinds of cool companies.
The last couple of months we’ve been working very hard at the next evolution of the Activiti engine. The core engine is undergoing a huge refactoring and we’ve got plenty of cool new stuff enabled by these changes up our sleeves. So after five years of version 5, we believe it is time for Activiti version 6!
We’re currently working on having all unit tests we have to run green on the Activiti 6 engine (almost there!). And yes, because that question will come first thing in the comments anyway, version 6 is fully compatible with version 5 (database schema, API’s, concepts, etc.).
We’re launching the new engine and unfolding all the core changes and plans at our Activiti Community Day in Paris (10th of June – only three weeks away). So let me repeat what I said in my previous post: If you are working with Activiti today, you don’t want to miss it. If you plan to use Activiti, you don’t want to miss it. It will be a major milestone and you will have the opportunity to witness it from the front line and influence it.
Attending the Community Day is easy, it’s completely free and you just need to register here: https://www.alfresco.com/events/global-activiti-user-day. Places are limited, so be quick!
Hope to see you there! And looking forward to show all the Activiti 6 sweetness!
As I wrote two days ago, we are organising an Activiti Community Day, 10th of June in Paris. In case you missed that post, read all about it here.
I’m very happy to announce that the registration is now live for this awesome event: http://www.alfresco.com/events/activiti-user-day . So register quickly and make sure your spot is reserved!
It’s been a while since the last one, but now it’s back (and how!): the Activiti Community Day!
And it’s going to be a Community Day that will go in the history books: not only have we arranged a superb location right in the heart of Paris (with a panoramic view over the city), we also have huge news which we will announce at the event. Activiti is now five years old, and it has grown more than we ever anticipated and it is used all over the globe in all kinds of industries and ways beyond our imagination. In those five years, we’ve learned a ton of how people use Activiti and how they want to use it going forward. So, in the past months, we’ve been working hard at the next evolution of the core Activiti Engine. And what better place to announce and show it than on the Activiti Community Day?
Do I hear mumbling of Activiti v6? No Comment. 😉
And if that wasn’t enough already, we’re currently lining up some top-notch speakers with real-life Activiti experience in very interesting environments. More about that soon once we have further fleshed out the agenda.
If you are working with Activiti today, you don’t want to miss it. If you plan to use Activiti, you don’t want to miss it. It will be a major milestone and you will have the opportunity to witness it from the front line and influence it. We don’t just plan to show it, we plan to really discuss the roadmap and ideas for the very future of Activiti!
Oh yeah, did I already mention it’s completely free? No entrance fee thanks to sponsoring from Alfresco!
So block your agenda and arrange your travel:
10th of June 2015, Espace Montmartre : 5, rue Saint Eleuthère 75018 Paris (very close to the Sacré-Cœur basilique)
Details of the venue: http://www.groupe-pearl.com/location-salles-paris/espace-montmartre. When there’s a registration page live, I’ll post a new blog.
Interested in doing a talk during the Community Day? Do reach out to us (here below this post, on twitter, email, …)! We still have a couple of slots open!
My good friend Josh Long did an interview with me about Activiti and Business Process Management in general. I must admit, I was quite nervous before the recording, as I had never done a podcast before (note : the editors at SE radio did a really great job 😉 ).
All feedback, as always, much appreciated!
With the Activiti 5.17.0 release going out any minute now, one of the things we did was writing down documentation on how to use this release together with Spring Boot. If you missed it, me and my Spring friend Josh Long did a webinar a while ago about this.
You can find the new docs already on github master (scroll down to ‘Spring Boot’ section): https://github.com/Activiti/Activiti/blob/master/userguide/src/en/ch05-Spring.adoc#spring-boot. You can also see we switched to Asciidoc for our docs a couple of weeks ago. And GitHub renders that natively, which is awesome.
While I was writing the documentation, I created a sample application to verify all the stuff in there actually works. You can find that example here: https://github.com/jbarrez/spring-boot-activiti-example. It has tags for each of the steps (which match the steps in the docs). So to start, checkout tag step-1 and so forth.
This is the first ‘official’ release of the integration (it was in snapshot versions before), so do give it a spin and let us know what you think about it!
Once every while, the question on how to create a pull request for Activiti is asked in our Forum. It isn’t hard to do, but specially for people that don’t know git it can be daunting.
So, I created a short movie that shows you how easy it is and what you need to do to:
Looking forward to your awesome fixes and features in a pull request!
Edit: Zoltan Alfattar mentioned to me on Skype there is a better way using branches:
@jbarrez Nice, however creating separate branch would be the better https://t.co/nco8rRZlk3
With which I agree fully. However, for people who are new to Git, it might be too complex to grasp. But surely have a look at the link if you are interested in doing it in the ‘proper’ way.