Hugo Troche's blog

Project Turnaround Specialist Challenge 1. The Project is Late.

I see this problem all the time. And I also see all the cliché knee jerk reactions to it: hiring temps (never works, Fred Brooks in “The Mythical Man Month” actually found out that adding resources in the middle of a software project slows the project down). Threating developers with dire consequences if they don't finish their tasks on time. And the crowd favorite, starting a death march with twice daily status meetings (one at 7:00 AM and the other at 7:00 PM) that everybody is required to attend.

If you are going to remember anything from this entry, remember this: Problems with the project schedule are the project's manager or team lead fault. You can't blame the team, can't blame vendors, can't blame senior management, can't blame other stakeholders. If you are in charge of the schedule it is your fault. Senior management will hold you and no one else accountable. Moving to the good news, you get a second chance to fix the project. If you read my last blog, you know you will not get a third.

This is what you need to do:

a) Find out how much work in relation to the total has been done. You need to be able to tell senior management with the backing of numbers that X % of the project is done. For example, if you are using Extreme Programming as your software process, you need to rate the user stories by difficulty and then calculate the teams user story velocity. User story A takes 1 unit of time, story B takes 2 units. Then if you are done with user story A, you are 33% done. This is the type of logic and numbers you will need to present to senior management to give them confidence on the state of the project.

How yo calculate where you are depends on your software process, historical data, etc. For example, you could a system like COCOMO II to get and estimate on time with an input of lines of code or function points. Again, this is not about how you calculate this number, there are countless methods, even I contributed to the literature with an article on how to use Design Patterns to Estimate lines of code in a project.

b) Once you know how much of the the project is done, create a new schedule with that number as a reference. The completion of the project will now be past the original deadline. With your numbers you can defend your position that the current scope will not be completed until the new date. If the stakeholders want a product by the original date the will have to reduce the scope.

This is the part in your negotiations that I call cliché storm. The stakeholders will come up with all kind of ideas on how to “motivate” the team to produce more. Be ready, read Fred Brooks, read “Death March”. In these books you will find suitable answers to most “motivational” techniques. Microsoft did a study to find out how many hours their software engineers really worked during the day. Because of the culture many engineers were at the office for 10 to 12 hours every day. What the study found out is that only 8 hours were actually productive, the rest of the time was used doing personal chores, gossiping, water cooler talk, etc. It is very difficult for an information worker like a software engineer to be productive for more than 8 hours a day. Find research like these on line and in books, be ready to show it.

If you can't sell the new schedule or reduced scope, your project is doomed. There is no “Magic Silver Bullet” (another great Fred Brooks book). New resources will not help, “motivation techniques” will not help (they will actually hurt because of turnover and poor quality to meet deadlines). You need to change the scope or the schedule.

Turning a project around

This is the first of what I hope a series of entries on practical advice on how to turn a software project around. Before I start on the subject let me tell you a story.

It is said that the great master of warfare Sun Tzu (author of the Art of War) was once trying to convince the Chinese emperor that he would be a great general to lead the emperor's troops. In order to prove that, Sun Tzu told the emperor that he could make the emperor's concubines behave like soldiers in a manner of minutes. To which the emperor laughed, the concubines did not even take the emperor's orders seriously. The emperor told Sun Tzu that if he could make his 100 concubines behave like soldiers, Sun Tzu will be the commanding general of the army.

Sun Tzu called out all the concubines, gave them spears, and told them to form in two columns of 50 concubines. At the front of each column, he placed the two favorite concubines of the emperor. He gave each of the two concubines the rank of lieutenant of the concubine army. Rank which the concubines whimsically accepted. Then he told the 100 concubines to face him while holding their spears in attack position. The concubines found this a lot of fun and followed his instructions. Then Sun Tzu grabbed a war drum and instructed the concubines to rotate 90 degrees to their right every time he beat the drum. At the first beat the concubines tried to rotate and started laughing when nobody got it right. Sun Tzu told the emperor that the first time troops make a mistake is usually the fault of the general. Therefore he would repeat the instructions and make the lieutenants responsible for the outcome. He explained the drill to the concubines again, and made a special mention to the lieutenants about their responsibility. So, Sun Tzu beat the drum again with the same results as before. The emperor and the court were truly enjoying the spectacle and laughing with the concubines.

This time Sun Tzu said that since he already repeated his order twice, that the fault was with the lieutenants. Therefore, he would cut the head of the most senior lieutenant off as an example to the others. The emperor jumped out of his seat and told Sun Tzu that he indeed appreciated Sun Tzu's wisdom and the first general opening will be Sun Tzu's. Sun Tzu told the emperor that training the concubines was his duty and that the emperor agreed to the task, therefore he should be given the liberty to finish the task. The emperor knew he had to let Sun Tzu finish. If he didn't, it will be known that the emperor went back on his word. Not a good reputation to have among generals. With the emperor looking by, Sun Tzu cut the head of the emperor's most beloved concubine. Then, he promoted the third most beloved concubine to lieutenant and told the second most beloved concubine that she was in charge. From then on the exercise went flawlessly. The emperor made Sun Tzu commanding general the army. Sun Tzu went on to win countless wars, many of them without ever fighting a battle.

The moral of the story is that if you are general, you should be ready to cut the head of careless lieutenants. If you are a lieutenant, you only get two chances.

I find this to be true with projects also. If you are managing a project, and the project gets in trouble, you will usually be given a chance in the form of time and resource to fix it. But, if you fail to fix it after that, well you read the story.

From the perspective of senior management a project is in trouble when:

a) The project is delayed and behind its planned schedule.

b) The project is costing more than planned.

c) The project is not delivering what the scope states it should.

If you want to become a software project turnaround specialist that walks in after some heads have rolled, or if you like your head attached to you shoulders you need to able to quantitatively identify to senior management why a), b), c) or a combination of the three is happening. Simple saying the schedule is too aggressive or I don't have enough resources will not do. You need numbers to back these claims. In my next post, I will explain how to use software duration estimation techniques to come up with a real time estimate for a project.

Single Sign On (SSO) in SOA

A common problem that I see in the field is how to enforce the security policy in the execution context of a service (are those enough SOA concepts in one sentence for you?). The market is full of Federated Identity Management tools. Once you decide on what identity management package you will use, you have the task to actually integrate that solution with the rest of your infrastructure.

Let's supposed that you decide to go with a SAML token solution from a vendor that is big and blue for example. The way this will work, is the identity management server will either intercept or redirect the initial communication going to your services, and challenge the service consumer to provide credentials. If the identity server accepts the credentials it will consider the service consumer authenticated and place a SAML token in the header of the request to the service.

Now, suppose the service the consumer is trying to access is a web service. What the identity server will do is place a SAML token in the SOAP header of the request going to the web service. All the web service container needs to do now is verify that the request has a valid SAML token. This would take care of authentication. By accepting the token the web services acknowledges that the consumer is who it pretending to be. The SAML authentication process is actually more involved than that, but for the example this will do.

If you are using JBoss, this is what you need to do to make your web services accept the token:

a) Create a generic handler (according to the JSR 109 specifications) that gets called before it ever hits the endpoint. You insert this handler in the chain. You make the handler deny the request if the
SAML token in the SOAP header is not valid.

b) You set the principal in the SOAP envelope to the principal that you read from the token. This will allow your JAAS login module to have a Subject loaded.

c) You create a custom JAAS login module that takes the Subject and performs authorization. Just because the consumer is who it say it is, it does not mean that it has access to this particular service. A common solution is to save the user/roles information for JAAS login module in an LDAP server. Then, the login module loads what roles are assigned to the user and the application server decides whether the role can access the resource or not. You define what roles can access a resource in the application configuration files.

This architecture will work with other type of tokens, or non token based federated identity management. In this particular case the authentication was separated from the authorization because of limitations with federated system. The SAML token can easily contain the role information for the user, so the application server does not have to hit the LDAP server as well.

Let me know if you need more details or have any questions, comments, etc.

Simplifying Deployment

A common problem that my customers face when developing and deploying Enterprise applications is how to keep all the different application server instances configurations synchronized. What I mean is, how to easily make sure that the development application server, the QA application server and the production application server have the same configuration, and when the development application server changes, what is the process to move those configurations changes to the other application servers. I am not talking about application code itself, but the configuration files and directories on the servers. I see this issue even in organization with a mature software process.

I have seen many solutions to this problem. Most of the time the solutions are depending on a technology, are manual and poorly documented. Here are some examples I have seen:

- README notes. I have to admit I have a soft spot for README notes. As a person heavily involved in Open Source, I am used to getting building, installing and upgrading instructions from README type documents. These notes alone just don't do the trick. They can be an important instrument in the process.

- Saving entire application server configuration directories in the repository. This one is a bad idea all around. For example, I have seen applications being moved from development to production by copying the entire server directory to production. To do this in JBoss, you would copy the directory //DevelopmentServer//%JBOSS_HOME/server/default to //ProductionServer//%JBOSS_HOME/server. This is bad because that instance of the server is only useful for the one application you are deploying. Deploying another application to that instance is difficult and complicated. Also, the possibility you will have broken file links is high.

As an enterprise architect the first thing I do to solve this problem is first to create a non technology specific strategy that can be instantiated to a particular technology stack. The plan will look like this:

a) Push as much configuration as possible to the application resource file (.ear, .war., .sar, etc).

b) Define what configurations can not be pushed to the application resource file. For example in JBoss that usually is : xxx-ds.xml, xxx-service.xml, login-config.xml among others.

c) Create a repository project separate from all the applications where you keep those files. Tell developer that their configurations have to play nice with everyone elses.

d) Create scrips with Ant, Maven, etc that automatically get those files from the repository and place in the correct place.

As a deliverable of this solution you should provide documentation of what can't be placed in the application resource file, a repository project for those files, scripts to automatically perform do the update to the servers from the repository.

Let me know if you have any questions.

AJAX and Messaging. The AJAX Pull Architecture Dilemma

With more and more AJAX applications popping up, I am seeing this and solving this problem often. It is well known in the thick client world that to keep thick clients over the network in the same state you have to use a Push Architecture with a messaging architecture in the back end. Let's take a look at the most common example on an application that needs to keep state synchronized across all clients, the Dashboard.

The Dashboard is usually a table (sometime something much more complicated) that contains information that is updated in real time by other clients and other services. For example, imagine a trucking company that has a table that contains all their trucks, who is driving their truck, what the truck is carrying, where the truck is going is going and the truck estimated time of arrival to its destination. This information is vital to the dispatchers to allow them to know what truck will be available when and where. Then they can make the most efficient use of the trucks on the road.

The trick is to keep this tables synchronized across all the clients without having the clients poll the database every x seconds to see if there are any changes. Such strategy simple does not scale. What you do in this case is you set up a messaging server in the back end like JBoss Messaging or TIBCO, create a topic in the Messaging server that all the clients subscribe to and then every change made to the table by clients, or maybe a GPS service, is published to the topic and all clients get a new message and update their table. This is a more efficient model.

How do you achieve a similar push architecture with AJAX? It is easy to create a dynamic AJAX table that updates itself asynchronously without ever resubmitting the page. You can do it with straight Javascript and HTML, or you can use a library like Google's GWT or DWR. The problem is that the AJAX page will eat up your server resources. By constantly polling for changes, you will have constant queries on your database when they are not necessary. But, how do you implement messaging with AJAX?

If you want to make your life easier, use Apache ActiveMQ as the JMS provider.
ActiveMQ provides Javascript remoting objects for the AJAX pages to listen for messages asynchronously. If you want to use another framework or JMS provider, you will need to create a POJO or servlet on the server side that writes and listens to the JMS topic, and the remoting Javascript objects need to listen to events from that POJO or servlet. JBoss is working on a similar feature for their JMS product. TIBCO also has AJAX libraries and support.

With all the messaging support for messaging and AJAX in the market, there is no excuse to use Pull Architectures in AJAX clients anymore.

Email me if you need more details.