Hugo Troche's blog

Why aren't Workflow engines parts of almost every JEE application? The Missed Opportunity

Workflow Engines like jBPM from JBoss are the perfect tool to model any process that has many paths, and requires human involvement between process tasks. Let's just start thinking about some common applications:

- Shopping cart: It is a perfect fit for a workflow engine. Each step in the process of buying an item is well defined, each step requires human interaction and finally there are many paths in the process (a shopper can pick from a myriad of shipping options for example, each with its own distinct actions to follow).
- Recruiting website: A candidate needs to enter his/hers skills. The system then matches the candidate to a job and notifies a recruiter and the candidate. The candidate and the recruiter review the match. If there is common interest, both parties communicate and so on.

I can keep going on and on with application examples that are simple business process representations. If that is the case, why don't we use JBPM or similar frameworks more often? Why are we not fluent in BPEL? After all, a workflow engine provides us with so many advantages:

a) It saves the state of the process. If a shopper exits a website before finishing with the shopping cart, the engine will easily bring the back session exactly to the point where it was before the user exiting. Implementing that logic without a workflow engine is usually time consuming and error prone.

b) It provides with a clear separation between the business process definition and the process task itself. jBPM is so effective in abstracting the task from the process that a non developer can reorganize the tasks later on. Most of these tools also have GUIs to manage the business processes. Try implementing that from scratch.

c) It reduces the complexity of applications by leveraging the engine to do the "plumbing" work of the business process. The developer just needs to focus on the process tasks and the business analyst in the process.

There are many other reasons about why a workflow engine is such a great and fundamental component to almost any application. Yet, why aren't workflow engines more popular? These questions nagged me for a while. I think there are 3 main reasons why workflow engines are not more popular:

1) The learning curve to be proficient in any of these frameworks is high. Thanks to frameworks such as Spring, developers want tools that they can start using right away after reading a few tutorials. jBPM is no such tool. This by itself should not be a reason for the unpopularity of workflow engines, but combined with,

2) There is no dominant player in the market. The market is fragmented. Don't get me wrong, there are many standards that most players are using, yet the standards are too low level for the skills to translate between frameworks.

3) Workflow engines live in the murky grayness between developers and business analysts. Some of the commercial workflow engines are geared and sold to business analysts. While other engines like JBPM are more geared towards developers. The successful engine has to be able to address both concerns.

The workflow engine will start becoming the Rosetta  Stone of modern enterprise systems. With the current SOAs Web Service access orchestration is already difficult. To take advantage of the long tail of possibilities in an SOA, a developer needs a solid way to model the business processes that access all those web services. Then, as the market moves towards ESBs, workflow engines and rule engines will be the glue of modern enterprise systems.

So, who will come up with the Spring of BPEL? How will come up with a powerful simple, open source framework that scales?

Project Turnaround Specialist Challenge 1. The Project is Late.

I see this problem all the time. And I also see all the cliché knee jerk reactions to it: hiring temps (never works, Fred Brooks in “The Mythical Man Month” actually found out that adding resources in the middle of a software project slows the project down). Threating developers with dire consequences if they don't finish their tasks on time. And the crowd favorite, starting a death march with twice daily status meetings (one at 7:00 AM and the other at 7:00 PM) that everybody is required to attend.

If you are going to remember anything from this entry, remember this: Problems with the project schedule are the project's manager or team lead fault. You can't blame the team, can't blame vendors, can't blame senior management, can't blame other stakeholders. If you are in charge of the schedule it is your fault. Senior management will hold you and no one else accountable. Moving to the good news, you get a second chance to fix the project. If you read my last blog, you know you will not get a third.

This is what you need to do:

a) Find out how much work in relation to the total has been done. You need to be able to tell senior management with the backing of numbers that X % of the project is done. For example, if you are using Extreme Programming as your software process, you need to rate the user stories by difficulty and then calculate the teams user story velocity. User story A takes 1 unit of time, story B takes 2 units. Then if you are done with user story A, you are 33% done. This is the type of logic and numbers you will need to present to senior management to give them confidence on the state of the project.

How yo calculate where you are depends on your software process, historical data, etc. For example, you could a system like COCOMO II to get and estimate on time with an input of lines of code or function points. Again, this is not about how you calculate this number, there are countless methods, even I contributed to the literature with an article on how to use Design Patterns to Estimate lines of code in a project.

b) Once you know how much of the the project is done, create a new schedule with that number as a reference. The completion of the project will now be past the original deadline. With your numbers you can defend your position that the current scope will not be completed until the new date. If the stakeholders want a product by the original date the will have to reduce the scope.

This is the part in your negotiations that I call cliché storm. The stakeholders will come up with all kind of ideas on how to “motivate” the team to produce more. Be ready, read Fred Brooks, read “Death March”. In these books you will find suitable answers to most “motivational” techniques. Microsoft did a study to find out how many hours their software engineers really worked during the day. Because of the culture many engineers were at the office for 10 to 12 hours every day. What the study found out is that only 8 hours were actually productive, the rest of the time was used doing personal chores, gossiping, water cooler talk, etc. It is very difficult for an information worker like a software engineer to be productive for more than 8 hours a day. Find research like these on line and in books, be ready to show it.

If you can't sell the new schedule or reduced scope, your project is doomed. There is no “Magic Silver Bullet” (another great Fred Brooks book). New resources will not help, “motivation techniques” will not help (they will actually hurt because of turnover and poor quality to meet deadlines). You need to change the scope or the schedule.

Turning a project around

This is the first of what I hope a series of entries on practical advice on how to turn a software project around. Before I start on the subject let me tell you a story.

It is said that the great master of warfare Sun Tzu (author of the Art of War) was once trying to convince the Chinese emperor that he would be a great general to lead the emperor's troops. In order to prove that, Sun Tzu told the emperor that he could make the emperor's concubines behave like soldiers in a manner of minutes. To which the emperor laughed, the concubines did not even take the emperor's orders seriously. The emperor told Sun Tzu that if he could make his 100 concubines behave like soldiers, Sun Tzu will be the commanding general of the army.

Sun Tzu called out all the concubines, gave them spears, and told them to form in two columns of 50 concubines. At the front of each column, he placed the two favorite concubines of the emperor. He gave each of the two concubines the rank of lieutenant of the concubine army. Rank which the concubines whimsically accepted. Then he told the 100 concubines to face him while holding their spears in attack position. The concubines found this a lot of fun and followed his instructions. Then Sun Tzu grabbed a war drum and instructed the concubines to rotate 90 degrees to their right every time he beat the drum. At the first beat the concubines tried to rotate and started laughing when nobody got it right. Sun Tzu told the emperor that the first time troops make a mistake is usually the fault of the general. Therefore he would repeat the instructions and make the lieutenants responsible for the outcome. He explained the drill to the concubines again, and made a special mention to the lieutenants about their responsibility. So, Sun Tzu beat the drum again with the same results as before. The emperor and the court were truly enjoying the spectacle and laughing with the concubines.

This time Sun Tzu said that since he already repeated his order twice, that the fault was with the lieutenants. Therefore, he would cut the head of the most senior lieutenant off as an example to the others. The emperor jumped out of his seat and told Sun Tzu that he indeed appreciated Sun Tzu's wisdom and the first general opening will be Sun Tzu's. Sun Tzu told the emperor that training the concubines was his duty and that the emperor agreed to the task, therefore he should be given the liberty to finish the task. The emperor knew he had to let Sun Tzu finish. If he didn't, it will be known that the emperor went back on his word. Not a good reputation to have among generals. With the emperor looking by, Sun Tzu cut the head of the emperor's most beloved concubine. Then, he promoted the third most beloved concubine to lieutenant and told the second most beloved concubine that she was in charge. From then on the exercise went flawlessly. The emperor made Sun Tzu commanding general the army. Sun Tzu went on to win countless wars, many of them without ever fighting a battle.

The moral of the story is that if you are general, you should be ready to cut the head of careless lieutenants. If you are a lieutenant, you only get two chances.

I find this to be true with projects also. If you are managing a project, and the project gets in trouble, you will usually be given a chance in the form of time and resource to fix it. But, if you fail to fix it after that, well you read the story.

From the perspective of senior management a project is in trouble when:

a) The project is delayed and behind its planned schedule.

b) The project is costing more than planned.

c) The project is not delivering what the scope states it should.

If you want to become a software project turnaround specialist that walks in after some heads have rolled, or if you like your head attached to you shoulders you need to able to quantitatively identify to senior management why a), b), c) or a combination of the three is happening. Simple saying the schedule is too aggressive or I don't have enough resources will not do. You need numbers to back these claims. In my next post, I will explain how to use software duration estimation techniques to come up with a real time estimate for a project.