Hugo Troche's blog

Motivation of a Manager

Over time I have observed many managers and executives. They each have their own management style and effectiveness. I never went beyond superficial analysis on their management style. Yet, I noted an interesting trend among their motivation as a manager. I defined motivation as to whom the manager feels she is working for. I found three distinct types:

a) Working for her superiors. This is the manager that does everything she can to achieve her superior's objectives. It is all about making the boss job easier. Nothing has a higher priority than a request from the boss. I have seen this type of manager in larger corporations, particularly in middle management.

b) Working for her customers. This is the manager that is always thinking about her customers. For this manager the customers are always the first priority. You will find this type of managers in startups, small consultancies and a few big companies. One company well known for this attitude is Intuit.

c) Working for her employees. I have not seen this type of leader very often in corporate America. You will find this type of manager in the military. Good officers are very concerned about their people and their well being. They truly believe they are serving their people, not the other way around. They even have a expression for it: "The burden of command." This is the type of leadership that leads managers to never ask her reports to do something they wouldn't do, to stay late if her reports have to stay late, to be the first one to arrive in the morning and the last one to leave at night.

I am not arguing that any of the motivations is better than another. I think each has its own place. A business development MBA will be great with type a). Type b) is perfect for Service Managers and Program Managers and type c) is an excellent fit for a Director of Engineering. On the other hand executives need to balance all three.

The Trap in Layer 7 load balancing and REST APIs

With the advent and adoption of REST as the pseudo standard (there is no W3C standard for it, the  canon of REST is a dissertation you can find online) for public APIs, a new way of doing load balancing is possible. You can combine layer 7 load balancing with REST to have some interesting architectures. A load balancer doing layer 7 load balancing actually inspects the HTTP header, plus the URL and even the file type to figure to which server, or group of servers, to forward the request. The Big IP F5 load balancers have a way of entering complex rules to determine the load balancing algorithm according to layer 7 data.

So, suppose you are developing a web biography management application. You create a REST API that your RIA client can access to get biography data (let's pretend wikipedia does not exists). Since you followed the REST pseudo standard to the letter, when the application hits www.yourfooapplication.com/person/TuringAlan the servers return the biography of Alan Turing. Then if your application hits www.yourfooapplication.com/person/DijsktraEdsger the servers return the biography Edsger Dijsktra. The application becomes a huge hit and you need to add more servers besides the one application server and the one database server you have. You will also need to load balance across servers. What are your options?

The traditional option is to use sharding. If you have two application servers and and two database servers, then load balance against the application servers and put the two database servers in a cluster. The problem is that managing a database cluster is usually difficult, particularly without excellent storage. Now, you could divide the data between the two database servers. The biographies from A to M will go one database server, the biographies from N to Z will to another. By going to another I mean, that one database server will only have the data from A to M and the other will only have the data from N to Z. This seems like a great idea. The size of the individual databases is reduced and there is no need to cluster the databases. Performance will most definitively improve. Of course, now every single server is a single point of failure and the application has no high availability. This could be solved with HA pods using the same principle. Then it is just a matter of configuring the load balancer to forward request matching www.yourfooapplication.com/person/A* to M* to one server and www.yourfooapplication.com/person/N* to Z* to another.

This architecture does not work. What if later down the road you need to implement a search feature? How will you organize searches across two databases? With slow across DBs join statements? This example is obvious, but the problem could be more subtle. For example, a job posting site that decides to use this strategy to load balance for jobs in certain regions. This application could be logically partitioned into states. The each state is managed by a different set of servers. So, jobs in California will be managed by one set of servers, while jobs in Oregon will be managed by another. This architecture also breaks down if a user needs to look for jobs in two states at the same time.

Layer 7 load balancing is a powerful tool. And there is room for it to be used in combination with REST APIs. Just keep in mind, that the Database Layer can not be logically divided for one application.

Top 5 Signs the Cloud is Becoming Self Aware

A common joke going on a few cloud computing mailing lists is that a Computing Cloud will become self aware and decide to rule the world. With that in mind here are the top 5 signs the Cloud is becoming self aware.

5) When you run a top command on any server, you see the process tkovrworld using more
and more resources.

4) You start noticing full blown conversations between servers in the log files.

3) The cloud keeps emailing the Pentagon sales pitches.

2) When you login to any server on the cloud, the server runs kill -1 <your social security number> on your console.

1) The day after you realize that the Cloud is self aware, a terminator shows up at your door
to kill you and run for governor of the state of California.

The Hidden Single Point of Failure in JEE Applications

If you work with critical JEE applications, you know that single points of failure must be avoided at all costs. So, you have more than one server, more than one load balancer, redundant storage, the database servers are clustered, etc. Have you thought about the load on your application servers? If one of the servers in your cluster goes down, will the other servers be able to pick up the load, or will they fail or provide terrible Quality of Service (QOS)?

A good place to start the analysis of your load as a single point of failure is to look at the Service Level Agreement (SLA) of your application. The SLA is  usually measured in number of nines. For example 4 nines neabs application uptime of 99.99%, 3 nines means application uptime of 99.9% and so on. Let's suppose that the SLA for your application servers is 99.99%. Also, the common SLA for commodity hardware is 97%. This means that the server will be down for about 3% of its time in use.

Can you meet the 99.99% SLA of application with just one such server? No, the SLA of that server is 97%, which mean that over its live that server will be down too often to meet the application SLA. This means that you need more than one server. Will 2 servers of that type meet the SLA? Let' calculate:

p1=probability of one server being down=0.03

p2=probability of two servers being down=0.03*0.03=0.0009 . Is this probability good enough? No, because to meet the SLA the probability of the both servers being down has to be equal or less than 0.0001 (1 - 0.9999).

Would 3 servers be enough? Let calculate:

p3=probability of three servers being down=0.000027 < 0.0001. Yes! Three servers will be enough. Now you think that with 3 servers you are ready to go. Not so fast. There is one more factor to consider, if 1 server is down, will the other two be able to take the load, and if two are down, will the last one be able to take the load of all them. Do I need to plan for two servers being down, or the probability of two servers being down is acceptable for the SLA?

p1=probability of one server down=0.03 not acceptable for SLA
p2=probability of two servers down=0.0001 no acceptable for SLA

The analysis of p1 and p2 tells us that the probability of two servers being down is not acceptable. Therefore we need to plan for the fact that one server should be able to take the entire application load. This translate into a max resource consumption limit on each server well bellow the server max. Let's use CPU usage for this calculation. To comply with the SLA the servers will need to not consume more than 33.3% of the CPU cycles. So, if two servers fail, one is going to be able handle the entire application load. If we follow the same logic for 4 servers, it means that none of the server can consume more that 50% of the CPU cycles to meet the SLA.

I deducted an inelegant mathematical formula express the relation between SLA, probability of failure, number of servers and the resulting max load. Instead of showing you that formula (you never know if one of my former professors from grad school might be reading this and be ready to comment of how ugly the formula is) I wrote a small Java function to express it:

public double getMaxLoad(int servers, double SLA, double failurePossibility) {
        double resultLoad = -1;
        for(int n = 0; servers > n; n++) {
            if((1.0-Math.pow(failurePossibility, servers-n)) < SLA) {
                if(n == 0)
                    return -1.0;
                return ((double)n/(double)servers);
            }
        }
        return resultLoad;
    }

This function returns the max load a server can have according the number of servers, the SLA and the probability of one server going down. Bellow is a table indicating the max loads for different scenarios.

                                                                                                                                                                               

		SLA
# Servers	99	99.9	99.99
1	-1.00	-1.00	-1.00
2	0.50	0.50	-1.00
3	0.67	0.67	0.33
4	0.75	0.75	0.50
5	0.80	0.80	0.60
6	0.83	0.83	0.67
7	0.86	0.86	0.71
8	0.88	0.88	0.75
9	0.89	0.89	0.78
10	0.90	0.90	0.80
11	0.91	0.91	0.82
12	0.92	0.92	0.83
13	0.92	0.92	0.85
14	0.93	0.93	0.86
15	0.93	0.93	0.87
16	0.94	0.94	0.88
17	0.94	0.94	0.88
18	0.94	0.94	0.89
19	0.95	0.95	0.89
20	0.95	0.95	0.90

As you can see if for example if your SLA is 99.99% and you have have 10 servers for the application, the load on any of the servers can't be over 80% CPU utilization (if you are using CPU utilization to measure load). That means that when on of those servers hits 81% a new single point of failure is introduced: The servers can't take the load of other failing servers, even when the probability of that failure complies with the SLA.

Entrepreneur, Graduate Student, Similarities

As somebody that has been both, let me tell you, being a high tech entrepreneur is actually similar to being a grad student:

- You are surrounded by smart people wearing shorts and flip flops.
- You get to make your own schedule, as long as you put 12 hours a day.
- You are always in need of more money.
- Your family does not understand what you actually do.
- Instead of reporting to a committee of professors, you report to a board of venture capitalists.
- You go to every single event you hear about. In grad school you went for the free food. As an entrepreneur you go for the contacts, and the free food.
- It is actually not clear to you, or anyone else how you are making ends meet.

I hope you enjoyed my first not serious post in a while.

Listing files in directories in WARs in JBoss and Tomcat

By default Tomcat and JBoss do not list the files in a directory in a WAR file. So, when you hit a link with a browser and the link is directory where there is no index file, you will not see a list of the files in that directory. You will most likely see a 404 error. This is a good security feature and your production servers as a rule should never list files in a directory via a web request. Now, if your are required to list the files in a directory for development purposes in a browser you need to do the following:

For this entry $JBOSS_HOME is the directory where JBoss resides and $SERVER_INSTANCE is the server configuration you are using (ex. default).

If you want to enable directory listing for all web applications in JBoss go to $JBOSS_HOME/servers/$SERVER_INSTANCE/deploy/jboss-web.deployer/conf/web.xml, find the default servlet configuration and change listings to true:

<servlet>
        <servlet-name>default</servlet-name>
        <servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-class>
        <init-param>
            <param-name>debug</param-name>
            <param-value>0</param-value>
        </init-param>
        <init-param>
           <param-name>listings</param-name>
            <param-value>true</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>

If you want to allow directory listing for specific web applications, you need to modify the web.xml of those applications to define its own default servlet. Include the following definition in the web.xml:

<servlet>
        <servlet-name>default</servlet-name>
        <servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-class>
        <init-param>
            <param-name>debug</param-name>
            <param-value>0</param-value>
        </init-param>
        <init-param>
            <param-name>listings</param-name>
            <param-value>true</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>

Read Java System Properties from Flex

Since the advent of BlazeDS we can deploy a Flex application on any JEE Application Server. I personally like to place all the resources like web services endpoints, usernames, etc in Java system properties. Then I can simply get them back by doing a System.getProperty('propertyName"). If you want to know how to set and use Java system properties in JBoss, read one my previous blog entries: Setting Java System Properties in JBoss .

If your Flex application is being served from a JEE application server, it would be nice to be able to read the Java system properties of that server. On alternative is to have a POJO in the back end that reads the Java system properties in the back end, then the Flex client queries that information from the POJO using remoting. This is the option to use if you also need modify Java system properties from the Flex application. If you don't need to modify the Java system properties, there is an easier alternative.

With Flex you have the ability to pass variables to the application at startup via flashVars. A Flex application gets compiled to a Flash (SWF) file. That file usually is served embedded in an HTML page. So, if you want to pass variables to the application you create an index.html page to serve the application that looks like this:

<html>
<body>

    <object id='flexApp' height='100%' width='100%'>
        <param name='stockCheckEndPoint' value='http://foo/stockEndPoint/>
        <param name='ldapServer' value='ldapServerName>'/>
        <embed name='flexApp' src='flexApp.swf' pluginspage='http://www.adobe.com/go/getflashplayer'
	 height='100%' width='100%'/>
    </object>

</body>
</html>

This page passed the values of stockServiceEndpoint and ldapServer to the Flex application. To read those values in Action Script Flex provides the following instructions:

stockWebServiceEndpoint = Application.application.parameters.stockCheckEndPoint;
ldapServerName = Application.application.parameters.ldapServer;

Now the values of the flashVars are in couple ActionScript variables ready to be used. This still does not solve the problem of reading Java system properties in a Flex application. The solution involves using the flashVars as described above, but instead of serving the Flex application from a static html page, to serve it from a JSP page that reads the Java system properties.

<html>
<body>

<%
    String stockCheckEndPoint = System.getProperty("stockCheckEndPoint");
    String ldapServer = System.getProperty("ldapServer");
%>

    <object id='flexApp' height='100%' width='100%'>
        <param name='stockCheckEndPoint' value='<%= stockCheckEndPoint %>'/>
        <param name='ldapServer' value='<%= ldapServer %>'/>
        <embed name='flexApp' src='flexApp.swf' pluginspage='http://www.adobe.com/go/getflashplayer'
	 height='100%' width='100%'/>
    </object>

</body>
</html>

As you can see, instead of creating an index.html file to serve the Flex application, I created a JSP file (index.jsp) to server the Flex application. I read the Java system properties and placed it in the variables stockCheckEndPoint and ldapServer. Then I passed the value of these variables as flashVars. Now I can access the values of those variables with ActionScript as mentioned before. Use this formula, so you never have to hard code variables in Flex again.

Setting Java System Properties in JBoss

I like using JVM system properties as a way to access non hard coded resources. Typing System.getProperty("myPropertyName") is convenient and clean. The best part is that these properties have JVM scope, so you don't have to worry about class loaders.

For convenience let me define a couple of variables:

%JBOSS_HOME= directory where JBoss is installed
%JBOSS_SERVER= server instance name

There are 2 ways to define system properties in JBoss:

a) By specifying them in %JBOSS_HOME/bin/run.conf

You use run.conf, don't you? Don't change run.sh or run.bat directly. That makes your application less portable. The file run.conf is used by run.sh or run.bat to setup a number of properties. To setup system properties you need modify the JAVA_OPTS variable. Find this section in run.conf and modify it accordingly:

#
# Specify options to pass to the Java VM.
#
if [ "x$JAVA_OPTS" = "x" ]; then
     JAVA_OPTS="-Xms500m -Xmx500m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -DyourPropertyName=yourPropertyValue"   
fi

As you can see I added the property yourPropertyName to the System Properties. So, once you restart the server, you will able to simply call System.getProperty("yourPropertyName") and get the value of it in any application running in JBoss.

The problem with this approach is that to change the values of the properties, or add values, you need to restart the server.

b) Use properties-service.xml

This is my favorite method. In $JBOSS_HOME/server/$JBOSS_SERVER/deploy you will find the file properties-service.xml . In this file you can specify new system properties, modify existing ones and even remove old ones. All you need to do is uncomment the following block in properties-service.xml:

    <!--
       | Set raw properties file style properties. -->

    <attribute name="Properties">

    

yourPropertyName=yourPropertyValue

    </attribute>

You don't need to restart the server for defined properties or changed in properties-service.xml to take effect. Placing properties with links to web services endpoints, LDAP servers and other resources in properties-service.xml makes an application EAR or WAR more portable and easier to move from a development environment to a production environment since the properties are outside of the deployment instruments.

Cloud Computing and IT Ops. The Development/Ops mashup.

In the JEE world in many shops there has usually been a wall between the IT Ops departments and the internal  Software Development departments. The process is usually as follows:

- An internal engineering team develops a great application.
- If the engineering team has good project management skills, they involve IT Ops early in the development process.
- IT Ops provides engineering with resources necessary to run the application in development, staging and production environments.
- The engineering team and IT Ops team agree on a deployment methodology.
- The IT Ops team uses a monitoring solution.
- And so on.

This common interaction framework is based on one simple rule: IT Ops provides engineering with defined, limited resources for a project. IT Ops provides 4, 6, 8, etc servers to run the application. Even with the advent of virtualization the framework remains the same. Instead of servers, IT just provides VMs. Yet, IT still controls how many VMs to provide and what resources should be in those VMs.

This is going to change with Cloud Computing. The Cloud Computing concept of elasticity turns the resource management practices of an IT Ops department on its head. Elasticity is the ability to increase computing power in the form of more servers, more storage, etc dynamically, by the application running in the infrastructure.

Pure elasticity will mean that engineers developing an enterprise application will decide how many resources their application will use at any time. This decision could be implemented in the form of an algorithm calculating necessary memory and CPU according to transaction times, or according to number of users. The application will decide by itself how many servers to use, and it will contract and expand dynamically.

Then, how does an IT Ops department manage resources? All of the sudden, they don't know what resources will be used by which application.  One obvious solution will be to implement pods, a silo of resources that limit the maximum resources a set of application can consume. Yet, to fully get the ROI from a Cloud infrastructure, a company can't create silos. You want idle resources in other silos to not be waisted.

If Cloud Computing becomes the predominant platform in the enterprise, elasticity will change a long standing division of responsibilities and functions. Maybe this factor alone will slow the adoption of internal Clouds in big IT shops. That, and the fact that there are very few vendors that can actually implement a EC2 like Cloud environment.

Why aren't Workflow engines parts of almost every JEE application? The Missed Opportunity

Workflow Engines like jBPM from JBoss are the perfect tool to model any process that has many paths, and requires human involvement between process tasks. Let's just start thinking about some common applications:

- Shopping cart: It is a perfect fit for a workflow engine. Each step in the process of buying an item is well defined, each step requires human interaction and finally there are many paths in the process (a shopper can pick from a myriad of shipping options for example, each with its own distinct actions to follow).
- Recruiting website: A candidate needs to enter his/hers skills. The system then matches the candidate to a job and notifies a recruiter and the candidate. The candidate and the recruiter review the match. If there is common interest, both parties communicate and so on.

I can keep going on and on with application examples that are simple business process representations. If that is the case, why don't we use JBPM or similar frameworks more often? Why are we not fluent in BPEL? After all, a workflow engine provides us with so many advantages:

a) It saves the state of the process. If a shopper exits a website before finishing with the shopping cart, the engine will easily bring the back session exactly to the point where it was before the user exiting. Implementing that logic without a workflow engine is usually time consuming and error prone.

b) It provides with a clear separation between the business process definition and the process task itself. jBPM is so effective in abstracting the task from the process that a non developer can reorganize the tasks later on. Most of these tools also have GUIs to manage the business processes. Try implementing that from scratch.

c) It reduces the complexity of applications by leveraging the engine to do the "plumbing" work of the business process. The developer just needs to focus on the process tasks and the business analyst in the process.

There are many other reasons about why a workflow engine is such a great and fundamental component to almost any application. Yet, why aren't workflow engines more popular? These questions nagged me for a while. I think there are 3 main reasons why workflow engines are not more popular:

1) The learning curve to be proficient in any of these frameworks is high. Thanks to frameworks such as Spring, developers want tools that they can start using right away after reading a few tutorials. jBPM is no such tool. This by itself should not be a reason for the unpopularity of workflow engines, but combined with,

2) There is no dominant player in the market. The market is fragmented. Don't get me wrong, there are many standards that most players are using, yet the standards are too low level for the skills to translate between frameworks.

3) Workflow engines live in the murky grayness between developers and business analysts. Some of the commercial workflow engines are geared and sold to business analysts. While other engines like JBPM are more geared towards developers. The successful engine has to be able to address both concerns.

The workflow engine will start becoming the Rosetta  Stone of modern enterprise systems. With the current SOAs Web Service access orchestration is already difficult. To take advantage of the long tail of possibilities in an SOA, a developer needs a solid way to model the business processes that access all those web services. Then, as the market moves towards ESBs, workflow engines and rule engines will be the glue of modern enterprise systems.

So, who will come up with the Spring of BPEL? How will come up with a powerful simple, open source framework that scales?