NoSQL experimentations with Activiti: A (very simple) Neo4J prototype

I’ve got this itch for a long time now to try and see how easy it is to write a BPM engine on a graph database such as Neo4J. After all, the data model fits perfectly, as business processes are graphs and executions of those processes basically boil down to keeping pointers to where you are in that graph. It just feels as a very natural fit.

So I spend some time implementing a prototype, which you can find on https://github.com/jbarrez/activiti-neo4j

The prototype contains some unit tests which execute simple BPMN 2.0 processes. I tried to be as close as possible to the Activiti concepts of services, commands, executions, JavaDelegates, etc. Currently covered:

  • start and end event
  • user task
  • service task
  • parallel gateway
  • setting/getting variables

Much more information can be found at the readme on https://github.com/jbarrez/activiti-neo4j

I must say that working with Neo4j and its graph model for processes feel very very natural. It really simplifies stuff a lot and I can feel great potenital. Feel free to clone the code and check the code, it’s a regular maven project.

Note: we’re not moving to Neo4J with Activiti. This is just an experiment for myself. Of course, if you want to contribute you’re more than welcome!

28 Comments

  1. David Bigelow July 5, 2013

    WOW – this looks like a GREAT idea!

  2. Mike Dias July 8, 2013

    AMAZING Joram!!! We are already using Neo4j database with complementary data for the processes!

    Will be great contribute with this project!!! 😀

  3. Serge July 9, 2013

    Great idea, Joram!
    Is it any option to create Neo4J model using web builder?
    Serge

  4. Joram Barrez July 10, 2013

    @Serge: If you mean with web builder the Activiti Modeler, then yes. The goal of the Neo4j prototype is indeed to execute BPMN 2.0 as-is.

  5. Markus Menner July 16, 2013

    That’s really good news!

    Did you look into OrientDB (http://www.orientdb.org) as an alternative to Neo4J? We use that in our current project. It’s really impressive in terms of features, simplicity and performance…

    We use OrientDB along with a RDBMS, which we need (only) for Activiti.
    Having a GraphDB BPMN 2.0 engine would make our life much easier (provided that the actual DB implementation is decoupled from the engine).

    Therefore: please continue with this Project 🙂

  6. Joram Barrez July 17, 2013

    @Markus: thanks for the pointer. I did remember checking orientdb, but not very in depth. I think I stopped looking after I discovered that transactions don’t span among nodes (but neither does neo4j). But it looks quite nice, from a technical point of view.

    Do you have any insight in neo4j vs orientdb?

  7. Markus Menner July 17, 2013

    I looked into both DBs but I didn’t create a comparison list. However, there are some on the Internet, I believe.
    OrientDB provides different APIs, ObjectDB, DocumentDB and GraphDB. Behind the scenes it is always a GraphDB.
    If you use the ObjectDB API it “feels” like a OODB.
    It has SQL for queries, ACID transactions, MVCC and even a JDBC driver for legacy connections.
    Distributed Transactions are on their roadmap.
    The license is Apache, a big advantage over Neo4J.
    The performance is great and after all for me it was a matter of which DB would “lie best in my hand”, wich was OrientDB.
    One disadvantage of OrientDB is the fact that the community is quite small. But it’s growing…

  8. Markus Menner August 1, 2013

    Any ideas how this experiment will continue?
    Are there serious plans to have the possibility to “hook up” NoSQL DBs to Activiti?

    Markus

  9. Joram Barrez August 1, 2013

    For the moment it will say an ‘experiment’, meaning that this is just me, doing a few hours here and there just because I like it.
    My main goal with this post was to ‘feel the market’ whether there is some interest and secondly to ‘claim the space’.
    So basically, nothing planned and nothing serious. For the moment.

    Of course, you are more than welcome to chime in on the project 😉

  10. Birger Zimmermann August 25, 2013

    Very promising approach! I’d be also very interested, if there are ideas to isolate the persistence in activiti (if possible), so that the datastore could be an implementation detail, which is choosable per project.

  11. kalyan February 14, 2014

    Please implement a graph database (a kind of NoSQL). This graph database
    should consist of nodes (with have properties) for entities and edges (which
    have single or multiple properties and can be directional or bidirectional) for
    relationships, and support node indexing and query. The query language has
    following keywords: START, MATCH, WHERE, RETURN, ORDER BY,
    AGGREGATE, SKIP, and LIMIT.

    Example Input:
    Node martin = graphDB.createNode();
    martin.setProperty(“name”, “Martin”);
    Node pramod = graphDB.createNode();
    pramod.setProperty(“name”, “Pramod”);
    Node barbara = graphDB.createNode();
    pramod.setProperty(“name”, “Barbara”);
    martin.createRelationshipTo(pramod, FRIEND, since = 1998);
    pramod.createRelationshipTo(martin, FRIEND, since = 1998);
    martin.createRelationshipTo(barbara, FRIEND);
    barbara.createRelationshipTo(martin, FRIEND);
    START node = Node:nodeIndex(name = “Barbara”)
    MATCH path = node-[relation*1..3]->related_node
    WHERE type(relation) = ‘FRIEND’
    RETURN related_node.name, length(path), relation.since;

  12. RJ March 20, 2014

    Is there any future plan in Activiti to also support any NoSQL DB for persistence?

  13. Joram Barrez April 1, 2014

    @RJ: nothing concrete on the roadmap, no. What would be the use case?

  14. SG April 18, 2014

    @Joram, When you say a use case, Are you asking about the large process data we may need to save ? or anything specific like why NoSql/Rdbms.
    Please post some points on why you think NoSql should be given a thought for BPM.

  15. Joram Barrez April 22, 2014

    @SG: no, not about the data. This specific implementation is noSQL, because Neo4j is a graph database.
    Since a process definition and instance is represented as a graph, it is a really good fit.

    There is one area where big data becomes interesting: history. It would be very good to dump all the audit data into a nosql store.

  16. Josh Long May 12, 2014

    This is fantastic! I can just imagine whole classes of queries that are screaming fast on Neo4j vs. traditional persistence technologies. I hope this gets further evolved.

  17. Oleg July 14, 2014

    Hi Joram! The approach you’ve introduced here looks like a really good idea and also a perfect fit for one of projects we’re working on at the moment. We’ve run several tests to compare performance of activiti-neo4j with standard Activiti. It looks like starting from just 500 workflow instances, it can take 10 times longer to complete a user task in each one of them when you’re using a relational DB comparing to a graph DB. So it’s really impressive and we’re now looking into writing a proper implementation of activiti-neo4j, replicating full functionality that standard Activiti provides. Would you be interested in this kind of contribution to your project?

  18. Joram July 14, 2014

    It sure sounds interesting, but I would be careful with such microbenchmarks too 😉

    But certainly, I’ve got nothing against an activiti-neo4j. How concrete are your plans? I for sure want to be involved somehow!

  19. Ilja H July 15, 2014

    Hi Joram,

    thank you for your quick responce. My name is Ilja and I am Oleg’s colleague, we work together on this project.

    Let us clarify our purposes and thoughts. We would like to use back-end functionality of Activiti with Neo4J, because it seems to be faster and more scalable. We explored your code at activiti-neo4j and activiti-engine projects. It seems, that Activiti engine is tightly coupled with MySQL database and you implemented some custom services and classes that are not implementing standard interfaces, but look similar. For example, RepositoryService, RuntimeService and so on… The right decision would be to rewrite all these classes in order to implement Activiti’s interfaces but it seems to be quite hard to do.

    Current activity-neo4j implementation depends on only few artifacts: activiti-bpmn-model and activiti-bpmn-converter. It feels that it would be easier to develop activiti-neo4j without deep integration with the Activiti engine. But in this case users would not be able to painlessly migrate from MySQL to Neo4j.

    Ideally, activiti-neo4j should look like an additional plugin for Activiti, where users could specify a persistance implementation just in a single config file. But there is a lot of work with that. “Quick and dirty” solution would be to develop activiti-neo4j as a separate Java-library. Could I ask you what do you think about it?

    Thank you very much.

  20. Joram July 17, 2014

    > It seems, that Activiti engine is tightly coupled with MySQL database and you implemented some custom services and classes that are
    > not implementing standard interfaces, but look similar.

    Indeed, Activiti is tightly coupled with relational databases.

    I hadn’t bothered with the interfaces yet, i first wanted to get something running.

    > The right decision would be to rewrite all these classes in order to implement Activiti’s interfaces but it seems to be quite hard to do.

    I think that those Service interfaces should be fine to implement with neo4j, i don’t see difficulties there.

    > It feels that it would be easier to develop activiti-neo4j without deep integration with the Activiti engine.

    I agree

    > But in this case users would not be able to painlessly migrate from MySQL to Neo4j.

    No, that is true. But maybe that’s not really a goal to have. It would make both projects very complex. As long as process definitions are runnable on both, I believe it’s fine.

    But I agree it would be very cool if those could be switched by configuration…

    Maybe it’s better to first start with implementing functionality, and worry about compatibility later on, once we know the internals of the new implementation we can make a better judgement.

    How would you like to progress with this? Do you want to use the activiti-neo4j repo?

  21. Ilja H July 17, 2014

    Thanks for your reply.

    At the moment we’ve cloned your repo to our internal server and commit to it. But we would be happy to send a patch (pull request) to you and join your GitHub activiti-neo4j repo if you don’t mind. Because it would be great if you follow our development.

    Now I try to rewrite all the services to implement standard interfaces. For example, RepositoryService (from activiti-neo4j) should implement the interface org.activiti.engine.RepositoryService. My goal is to make other Activity modules to work with Neo4j. For example, activity-rest, that we plan to use in our project to provide an API to external clients. As far as I understood from the source code, the persistence layer relies on a service that ProcessEngine supplies to modules, for example:


    ActivitiUtil.getTaskService().someMethod();

    Now I try to return the neo4j implementation of the org.activiti.neo4j.services.TaskService from the method getTaskService() instead of org.activiti.engine.impl.TaskServiceImpl and make sure that the whole functionality works well.

    Let us know your thoughts.

  22. Ilja H July 17, 2014

    Additionally, we upgraded the project to use Neo4j version 2.1.2, because it has a lot of improvements since the release of version 2.

  23. Joram July 21, 2014

    > At the moment we’ve cloned your repo to our internal server and commit to it. But we would be happy to send a patch (pull request) to > you and join your GitHub activiti-neo4j repo if you don’t mind. Because it would be great if you follow our development.

    Whatever works for you. I’ve got no problems with giving you direct access.
    What do you prefer?

    > Now I try to return the neo4j implementation of the org.activiti.neo4j.services.TaskService from the method getTaskService() instead > of org.activiti.engine.impl.TaskServiceImpl and make sure that the whole functionality works well.

    Sounds good. If the code is on github, I can check it out!

    > Additionally, we upgraded the project to use Neo4j version 2.1.2, because it has a lot of improvements since the release of version 2.

    Sure, perfect!

  24. Ilja H July 23, 2014

    Thank you for your answers.

    Finally, my code is compilable, all the tests are green and now I can show you our fork with preliminary results. So, here it is:
    https://github.com/p-i/activiti-neo4j/tree/refactor-services-with-standard-interfaces

    I tried to implement all the Services as close to original Activiti as I could. Thus, this code works for deploying:


    processEngine.getRepositoryService()
    .createDeployment()
    .name("expense-process.bar")
    .addClasspathResource("/tests.bpmn")
    .deploy();

    and this code works to retrieve the list of all the assignees:


    List tasks = processEngine.getTaskService()
    .createTaskQuery()
    .taskAssignee("kermit")
    .list();

    These code snippets work both with MySQL and Neo4j.

    If you have any comments, criticism or thoughts, we are always open for discussion (let’s move our conversation to the GitHub). My next target is to create more realistic experiments and to implement all the functionalities, that are left undone.

    Furthermore, I decided to not create any hacks in the code and I modified the activiti-engine code a little bit. I just exctracted the interface for ProcessEngineConfiguration and all the code refers to this class throuhg the interface. It allows me to inject another implementation of a configurator. All the tests are green.

    Here is our fork:
    https://github.com/p-i/Activiti/
    and this is my patch for the described modification. Please, check it out:
    https://github.com/p-i/Activiti/compare/generalise-abstract-layers
    If you don’t mind, I would send it to Activiti team as a pull request.

    Looking forward to hear from you soon.

  25. Ilja H August 15, 2014

    Hello Joram,

    I just want to notify you that we successfully integrated Spring Data Neo4j framework to activiti-neo4j. Thus, all the necessary objects are managed by Spring the context. The code becomes closer and closer to the Activiti Engine codebase.

    Next step is to clean a code and perhaps to write a documentation. Then we will try to implement all the missing functionality.

    It would be great to get any feedback from you. Thank you very much.

  26. […] NoSQL Experimentations with Activiti: A Neo4j Prototype by Jorem Barrez […]

  27. Jeff Jacobson February 13, 2015

    Jorem,

    Can you offer a status update? I’m doing some research on BPM solutions and thought this looked really interesting. Will the next release of activiti support neo4j? How close do you think you are?

    Thanks,

    Jeff

  28. Joram Barrez February 13, 2015

    @Jeff: I’m nowhere close at all :p This was just a prototype … nothing more. It would take a lot of effort to get all activiti working on Neo4j.

Leave a Reply

Your email address will not be published. Required fields are marked *