Lightweight Approaches to End to End Process Orchestration

Flexible tools support the right abstractions

Once you realize the power of representing your business as a set of processes, you’ll want to find software to represent those processes — and run them. And there’s good news! Such software exists, and it is quite good. It is natural to want to take this further — all the way to end-to-end process orchestration. Not just the “Order” process. Or the “fulfillment” process. It is natural to want to go even further: the “Order to Fulfillment” process; or the “Order, to Fulfillment, to Cash, to Returns” process.

As you would expect, all the process modeling and execution software is pretty good at representing simple processes — our process versions of “hello world”. They’re also usually good at representing somewhat more comprehensive processes.

Clean Abstractions and End-to-End Process

And you’ll be relieved to learn that you can do this with BPM suites — or with open source solutions like Camunda. In fact we’ve both modeled and built processes with this kind of comprehensive view at BP3. And we’ve also built solutions that just address one piece of a larger end-to-end process (e.g. “Reverse Logistics”).

When you use a BPM Suite, it needs to be in control of every element of that end-to-end process in some fashion. While it may be possible to use it in more of a “listening mode”, that isn’t the natural way to leverage a BPM Suite. The control that these suites require includes requiring you to leverage the legacy codebase and scripting decisions that are baked into the designs of the software. So you may have to use their UI frameworks, or their rules language, or their scripting languages, or their databases, or their integration frameworks. Likely you will have to use all of the above. And you will need to use their IDE (Integrated Development Environment). On some platforms you have to use their versioning control system as well.

There is another way, however. Open source process engines take a more forgiving and flexible approach. Camunda specifically takes an approach that allows for end-to-end process orchestration without requiring that Camunda control everything about your process execution. This means that Camunda can play a small part or a big part in the end to end orchestration of work. It takes a lighter touch — and therefore doesn’t need to be in control at a detailed level, of every step within the process. And it offers many choices for synchronizing tech stacks with the other systems involved in the orchestration…

End to End Processes aren’t just Theoretical

This isn’t just a theoretical problem, however (and apologies, I cannot recall from which talk or article I jotted down this note, but if you recognize your own words let me know so that I can properly attribute the comment!):

Traditional BPM suites are designed with the assumption that most or all of a process is executed by the suite itself. However, core business processes span many different systems and endpoints: APIs, RPA bots, legacy user interfaces, IoT devices, AI/ML software, and more. Traditional BPM suites cannot effectively automate these processes at scale.

1. Think Big. Do the work to build the models of your end-to-end process and create buy-in in your organization for thinking through some of the high-level design decisions.
2. Do the Work. In parallel — pick a specific problem and solve it — in production. Leverage Camunda or another process engine.

If you try to leverage traditional BPM suites to automate processes that include considerable use of existing, legacy, or walled-garden systems — you start to find the conflicts between these systems as a real barrier and point of friction in implementation. Pressure builds to either replace the legacy system or to stop at process boundaries that that application defines.

In this context, we will generally recommend leveraging an open source process engine and platform to connect processes that span systems. This lets us weave together the functionality you need to build a more agile digital business — while minimizing the number of forced technical decisions, and points of friction.

Do These Two Things

If you are considering solving end-to-end process orchestration — and you’ve not previously worked with process engines and process-led development methods, I have one more recommendation: do two things at once.

Why these two things? Because you need to keep the big picture imperatives in mind — that thread of working on that picture will keep you grounded. But, you can’t just get lost with your head in the clouds about whiteboard diagrams. The discipline of building a software process in production will help ensure that you’re building the skills and organizational muscle memory to succeed as you take on more and more of your vision.

Quite simply, you want your organization to be great at the philosophical and the practical — and to do that you have to do both.

Originally published at https://sfrancisatx.substack.com on June 6, 2022.