Hugo Troche's blog

The Trap in Layer 7 load balancing and REST APIs

With the advent and adoption of REST as the pseudo standard (there is no W3C standard for it, the  canon of REST is a dissertation you can find online) for public APIs, a new way of doing load balancing is possible. You can combine layer 7 load balancing with REST to have some interesting architectures. A load balancer doing layer 7 load balancing actually inspects the HTTP header, plus the URL and even the file type to figure to which server, or group of servers, to forward the request. The Big IP F5 load balancers have a way of entering complex rules to determine the load balancing algorithm according to layer 7 data.

So, suppose you are developing a web biography management application. You create a REST API that your RIA client can access to get biography data (let's pretend wikipedia does not exists). Since you followed the REST pseudo standard to the letter, when the application hits www.yourfooapplication.com/person/TuringAlan the servers return the biography of Alan Turing. Then if your application hits www.yourfooapplication.com/person/DijsktraEdsger the servers return the biography Edsger Dijsktra. The application becomes a huge hit and you need to add more servers besides the one application server and the one database server you have. You will also need to load balance across servers. What are your options?

The traditional option is to use sharding. If you have two application servers and and two database servers, then load balance against the application servers and put the two database servers in a cluster. The problem is that managing a database cluster is usually difficult, particularly without excellent storage. Now, you could divide the data between the two database servers. The biographies from A to M will go one database server, the biographies from N to Z will to another. By going to another I mean, that one database server will only have the data from A to M and the other will only have the data from N to Z. This seems like a great idea. The size of the individual databases is reduced and there is no need to cluster the databases. Performance will most definitively improve. Of course, now every single server is a single point of failure and the application has no high availability. This could be solved with HA pods using the same principle. Then it is just a matter of configuring the load balancer to forward request matching www.yourfooapplication.com/person/A* to M* to one server and www.yourfooapplication.com/person/N* to Z* to another.

This architecture does not work. What if later down the road you need to implement a search feature? How will you organize searches across two databases? With slow across DBs join statements? This example is obvious, but the problem could be more subtle. For example, a job posting site that decides to use this strategy to load balance for jobs in certain regions. This application could be logically partitioned into states. The each state is managed by a different set of servers. So, jobs in California will be managed by one set of servers, while jobs in Oregon will be managed by another. This architecture also breaks down if a user needs to look for jobs in two states at the same time.

Layer 7 load balancing is a powerful tool. And there is room for it to be used in combination with REST APIs. Just keep in mind, that the Database Layer can not be logically divided for one application.

The Hidden Single Point of Failure in JEE Applications

If you work with critical JEE applications, you know that single points of failure must be avoided at all costs. So, you have more than one server, more than one load balancer, redundant storage, the database servers are clustered, etc. Have you thought about the load on your application servers? If one of the servers in your cluster goes down, will the other servers be able to pick up the load, or will they fail or provide terrible Quality of Service (QOS)?

A good place to start the analysis of your load as a single point of failure is to look at the Service Level Agreement (SLA) of your application. The SLA is  usually measured in number of nines. For example 4 nines neabs application uptime of 99.99%, 3 nines means application uptime of 99.9% and so on. Let's suppose that the SLA for your application servers is 99.99%. Also, the common SLA for commodity hardware is 97%. This means that the server will be down for about 3% of its time in use.

Can you meet the 99.99% SLA of application with just one such server? No, the SLA of that server is 97%, which mean that over its live that server will be down too often to meet the application SLA. This means that you need more than one server. Will 2 servers of that type meet the SLA? Let' calculate:

p1=probability of one server being down=0.03

p2=probability of two servers being down=0.03*0.03=0.0009 . Is this probability good enough? No, because to meet the SLA the probability of the both servers being down has to be equal or less than 0.0001 (1 - 0.9999).

Would 3 servers be enough? Let calculate:

p3=probability of three servers being down=0.000027 < 0.0001. Yes! Three servers will be enough. Now you think that with 3 servers you are ready to go. Not so fast. There is one more factor to consider, if 1 server is down, will the other two be able to take the load, and if two are down, will the last one be able to take the load of all them. Do I need to plan for two servers being down, or the probability of two servers being down is acceptable for the SLA?

p1=probability of one server down=0.03 not acceptable for SLA
p2=probability of two servers down=0.0001 no acceptable for SLA

The analysis of p1 and p2 tells us that the probability of two servers being down is not acceptable. Therefore we need to plan for the fact that one server should be able to take the entire application load. This translate into a max resource consumption limit on each server well bellow the server max. Let's use CPU usage for this calculation. To comply with the SLA the servers will need to not consume more than 33.3% of the CPU cycles. So, if two servers fail, one is going to be able handle the entire application load. If we follow the same logic for 4 servers, it means that none of the server can consume more that 50% of the CPU cycles to meet the SLA.

I deducted an inelegant mathematical formula express the relation between SLA, probability of failure, number of servers and the resulting max load. Instead of showing you that formula (you never know if one of my former professors from grad school might be reading this and be ready to comment of how ugly the formula is) I wrote a small Java function to express it:

public double getMaxLoad(int servers, double SLA, double failurePossibility) {
        double resultLoad = -1;
        for(int n = 0; servers > n; n++) {
            if((1.0-Math.pow(failurePossibility, servers-n)) < SLA) {
                if(n == 0)
                    return -1.0;
                return ((double)n/(double)servers);
            }
        }
        return resultLoad;
    }

This function returns the max load a server can have according the number of servers, the SLA and the probability of one server going down. Bellow is a table indicating the max loads for different scenarios.

                                                                                                                                                                               

		SLA
# Servers	99	99.9	99.99
1	-1.00	-1.00	-1.00
2	0.50	0.50	-1.00
3	0.67	0.67	0.33
4	0.75	0.75	0.50
5	0.80	0.80	0.60
6	0.83	0.83	0.67
7	0.86	0.86	0.71
8	0.88	0.88	0.75
9	0.89	0.89	0.78
10	0.90	0.90	0.80
11	0.91	0.91	0.82
12	0.92	0.92	0.83
13	0.92	0.92	0.85
14	0.93	0.93	0.86
15	0.93	0.93	0.87
16	0.94	0.94	0.88
17	0.94	0.94	0.88
18	0.94	0.94	0.89
19	0.95	0.95	0.89
20	0.95	0.95	0.90

As you can see if for example if your SLA is 99.99% and you have have 10 servers for the application, the load on any of the servers can't be over 80% CPU utilization (if you are using CPU utilization to measure load). That means that when on of those servers hits 81% a new single point of failure is introduced: The servers can't take the load of other failing servers, even when the probability of that failure complies with the SLA.

Why aren't Workflow engines parts of almost every JEE application? The Missed Opportunity

Workflow Engines like jBPM from JBoss are the perfect tool to model any process that has many paths, and requires human involvement between process tasks. Let's just start thinking about some common applications:

- Shopping cart: It is a perfect fit for a workflow engine. Each step in the process of buying an item is well defined, each step requires human interaction and finally there are many paths in the process (a shopper can pick from a myriad of shipping options for example, each with its own distinct actions to follow).
- Recruiting website: A candidate needs to enter his/hers skills. The system then matches the candidate to a job and notifies a recruiter and the candidate. The candidate and the recruiter review the match. If there is common interest, both parties communicate and so on.

I can keep going on and on with application examples that are simple business process representations. If that is the case, why don't we use JBPM or similar frameworks more often? Why are we not fluent in BPEL? After all, a workflow engine provides us with so many advantages:

a) It saves the state of the process. If a shopper exits a website before finishing with the shopping cart, the engine will easily bring the back session exactly to the point where it was before the user exiting. Implementing that logic without a workflow engine is usually time consuming and error prone.

b) It provides with a clear separation between the business process definition and the process task itself. jBPM is so effective in abstracting the task from the process that a non developer can reorganize the tasks later on. Most of these tools also have GUIs to manage the business processes. Try implementing that from scratch.

c) It reduces the complexity of applications by leveraging the engine to do the "plumbing" work of the business process. The developer just needs to focus on the process tasks and the business analyst in the process.

There are many other reasons about why a workflow engine is such a great and fundamental component to almost any application. Yet, why aren't workflow engines more popular? These questions nagged me for a while. I think there are 3 main reasons why workflow engines are not more popular:

1) The learning curve to be proficient in any of these frameworks is high. Thanks to frameworks such as Spring, developers want tools that they can start using right away after reading a few tutorials. jBPM is no such tool. This by itself should not be a reason for the unpopularity of workflow engines, but combined with,

2) There is no dominant player in the market. The market is fragmented. Don't get me wrong, there are many standards that most players are using, yet the standards are too low level for the skills to translate between frameworks.

3) Workflow engines live in the murky grayness between developers and business analysts. Some of the commercial workflow engines are geared and sold to business analysts. While other engines like JBPM are more geared towards developers. The successful engine has to be able to address both concerns.

The workflow engine will start becoming the Rosetta  Stone of modern enterprise systems. With the current SOAs Web Service access orchestration is already difficult. To take advantage of the long tail of possibilities in an SOA, a developer needs a solid way to model the business processes that access all those web services. Then, as the market moves towards ESBs, workflow engines and rule engines will be the glue of modern enterprise systems.

So, who will come up with the Spring of BPEL? How will come up with a powerful simple, open source framework that scales?

EhCache vs Memcached caching for JEE Applications

Caching is a building block of modern JEE applications. In order to handle any significant load you will need to establish a 2nd level cache. Let's just clear a common confusion. A 2nd level cache is not the same as the cache implemented by databases. Those are 1st level caches and they usually cache data blocks, not specific objects. 2nd level cache is the cache implemented outside of the database.

In the JEE world caching is specified by the the JCache JSR. The two major open source providers of caching are TreeCache from JBoss and EhCache. Both allow the creation of a distributed cache and provide built in mechanisms to deal with dirty data. Also, both libraries work with Hibernate as 2nd level cache providers.

I have had good experiences with both libraries as an architect. The overhead of both libraries is small and they both implement JCache making them interchangeable. Both, EhCache and TreeCache work in the same JVM as the application server. Surprisingly, in the last few months I have encountered more and more JEE applications using Memcached instead of EhCache or TreeCache.

Memcached is a C++  program that acts as a 2nd level cache. It is the default standard for caching in the LAMP stack. Wikipedia and Facebook base their entire caching infrastructure on Memcached. Besides that, Memcached has APIs for PHP, Perl, Ruby and of course Java. Memcached can also work as cache inside MySQL to keep tables in memory. The Java APIs are pure Java APIs. Memcached also has a distributed mechanism implemented via smart hashing.

I can understand using Memcached in a JEE application to access Memcached infrastructure that many LAMP applications are using. But, to use Memcached as the first option for a JEE application just seems strange to me for the following reasons:

• The Memcached Java APIs are not JCache compliant. Which means you can't change your mind about them later on without paying a major price.
• The application server needs to connect to the Memcached process via sockets calls. This requires the serialization and transmission of objects between processes. Something avoided with the Cache being inside the application server.
• The JCache solutions just appear to be much faster with Java.

In order to see if my last point was valid I ran a small test. In a Linux dual core box with 4 Gigs of memory, I created a small Java application that tested both options. The application is a just a simple engine that puts and gets objects from the cache. The objects that it puts are a simple class representing personal information.

Memcached was running in the same machine as the Java program doing the testing. I first started by testing 10000 puts and 10000 gets. Then I moved to 20000 puts and 20000 gets. I kept doing the same until I got to 100000 puts and gets. I measured the average time in milliseconds to do 10000 puts and 10000 gets from the cache. I compared the results of the Memcached test against running the same test for EhCache. The object I was placing in the cache was exactly the same in both cases, and the engine did not know what type of cache it was using due to an abstraction. I used the spymemcache Java API for Memcached.

These are the results. The numbers represent the average time it takes to perform 10000 puts or 10000 gets in milliseconds.

	EhCache	Memcached
put	31	545
get	16	2072

As we can see EhCached is one order of magnitude faster for put operations and two orders of magnitude faster get operations.

I was planning to do a more complete test using various caching servers to test the distributed capabilities of each solution. But, after the results I decided I had enough information. I believe that we should not use Memcached for a JEE application unless, there is a necessity to interact with an existing Memcached cache.

Please email me if you want the source code of my tests.

Links

EhCache

Memcached

Java API for Memcached   

Push Architectures with Flex

One of my favorite subjects to preach about is the importance of designing push architectures in enterprise systems. The classic example of when a push architecture is needed is the dashboard. In order to keep a dashboard up to date the options are to have the client program regularly poll the server for changes or to have the server push updated information to the dashboard when necessary. The later alternative performs and scales better.

The problem is how to implement such architecture with web clients. A server can’t push data to web clients due to the stateless nature of http. The market has come up with a number of solutions to this problem. I talked about such solutions in an earlier post. The reason I am going back to this subject is due to new developments in Flex.

Flex is the form based cousin of Flash. It is arguably the most complete RIA framework in the market. Quick disclosure, I started working for Adobe a few days ago. Yet, I used Flex extensively in other projects before. There are two developments that made in my opinion Flex an attractive option for push architectures. First, last year Flex itself became Open Source. That means the Flex SDK is available for anybody to use at no cost. Second, in February this year, the Flex server product became Open Source in the form of BlazeDS. With BlazeDS you have access to powerful remoting capabilities. With BlazeDS a Flex application can access POJOs directly and most importantly it can subscribe to topics on the server.

BlazeDS is deployed as a simple war file in any JEE compliant server. The topics are defined in an XML configuration file. Then the Flex client simply subscribes to the topic by becoming a consumer with two lines of code, then every time that that the topic has a new message, it gets “pushed” to the Flex clients. In reality no message gets truly pushed to a client due to the nature of http. Flex just implement a very clever polling mechanism. You can see how to do all this in many tutorials online.

The flex topics can also be configured to listen to JMS topics/queues. This means that you can actually easily access and manipulate backend integration infrastructure like Tibco. I think that now Flex has become a must tool for any web developer.

The Essence of the Software Architect

I have been asked many times about the role of a Software Architect. I have been asked about the skills necessary to become a Software Architect. I think today in the industry there is a firm understanding of the technical skills necessary to be a Software Architect. I think that technical skills represent only 20% of what an effective Software Architect needs. The list usually includes:

-          - Years of experience as a Software Engineer.

-          -  Deep technical knowledge of relevant technologies to the position. Software Architects are comfortable reading formal specifications for new technologies like JSRs.

- -   - Mastery of Object Oriented design patterns and Enterprise design patterns.

- -   - Master knowledge of software process.

- -   - Solid knowledge of deployment and operations issues.

 

After being a Software Architect and seeing other Software Architects at work, bellow is my distillation of what I think constitutes the basic attitude of a Software Architect.

1) A Software Architect is a leader before an Engineer

It is much more important to have the buy in of all the engineers for a particular design, than having the perfect design. Driving the process for the buy in is the Software Architect responsibility. The Software Architect needs the technical skills to weight design decisions and explain both sides. Yet, if the Software Architect can’t transform his technical knowhow into agreement from his team members, he might as well not be able to write a hello world class. The world is full of great software architectures and frameworks that nobody uses.

2) A Software Architect does much more giving than enforcing

A very experienced Software Architect friend of mine once told me: “If I have to point a design flaw in an implemented system, I did not do my job well.” It is easy to look at an application after the fact and pointing problems and asking for changes. The effective Software Architect mentors the development teams way before a project starts about design patterns, consequences of picking certain technology components, tradeoffs, maintenance challenges, performance objectives, etc. He holds regular formal mentoring sessions in the shape of lunch and learn or similar activities. He continuously performs code reviews and mentors developers from what he sees there. He makes his first priority to always have time to answer developer’s questions, everything else is less important.

3) A Software Architect thinks globally but acts locally

It is important for a Software Architect to think of the implications that design decisions will have in the entire portfolio of applications and infrastructure. Yet, an effective Software Architect never starts a project, sets up infrastructure or writes code that does not implement a specification or feature of an application. He designs his applications to create an architecture pattern and framework, not a framework to create applications.

4) A Software Architect thinks in simple terms

An effective software Architect that I met in a consulting engagement always analyzed everything through a simple prism. He thought the implications of technical decisions in regards to usability, reliability, scalability and maintainability.  Whenever someone asked him to analyze a tradeoff, he always touched those four subjects. Everyone knew how he thought. A developer with a good idea would usually think about those four areas before coming to him. The parameters a Software Architect uses might be different from my friends, but as long as they are simple, consistent and well known they will have the same effect of getting the organization considering those parameters.

Single Sign On (SSO) in SOA

A common problem that I see in the field is how to enforce the security policy in the execution context of a service (are those enough SOA concepts in one sentence for you?). The market is full of Federated Identity Management tools. Once you decide on what identity management package you will use, you have the task to actually integrate that solution with the rest of your infrastructure.

Let's supposed that you decide to go with a SAML token solution from a vendor that is big and blue for example. The way this will work, is the identity management server will either intercept or redirect the initial communication going to your services, and challenge the service consumer to provide credentials. If the identity server accepts the credentials it will consider the service consumer authenticated and place a SAML token in the header of the request to the service.

Now, suppose the service the consumer is trying to access is a web service. What the identity server will do is place a SAML token in the SOAP header of the request going to the web service. All the web service container needs to do now is verify that the request has a valid SAML token. This would take care of authentication. By accepting the token the web services acknowledges that the consumer is who it pretending to be. The SAML authentication process is actually more involved than that, but for the example this will do.

If you are using JBoss, this is what you need to do to make your web services accept the token:

a) Create a generic handler (according to the JSR 109 specifications) that gets called before it ever hits the endpoint. You insert this handler in the chain. You make the handler deny the request if the
SAML token in the SOAP header is not valid.

b) You set the principal in the SOAP envelope to the principal that you read from the token. This will allow your JAAS login module to have a Subject loaded.

c) You create a custom JAAS login module that takes the Subject and performs authorization. Just because the consumer is who it say it is, it does not mean that it has access to this particular service. A common solution is to save the user/roles information for JAAS login module in an LDAP server. Then, the login module loads what roles are assigned to the user and the application server decides whether the role can access the resource or not. You define what roles can access a resource in the application configuration files.

This architecture will work with other type of tokens, or non token based federated identity management. In this particular case the authentication was separated from the authorization because of limitations with federated system. The SAML token can easily contain the role information for the user, so the application server does not have to hit the LDAP server as well.

Let me know if you need more details or have any questions, comments, etc.

AJAX and Messaging. The AJAX Pull Architecture Dilemma

With more and more AJAX applications popping up, I am seeing this and solving this problem often. It is well known in the thick client world that to keep thick clients over the network in the same state you have to use a Push Architecture with a messaging architecture in the back end. Let's take a look at the most common example on an application that needs to keep state synchronized across all clients, the Dashboard.

The Dashboard is usually a table (sometime something much more complicated) that contains information that is updated in real time by other clients and other services. For example, imagine a trucking company that has a table that contains all their trucks, who is driving their truck, what the truck is carrying, where the truck is going is going and the truck estimated time of arrival to its destination. This information is vital to the dispatchers to allow them to know what truck will be available when and where. Then they can make the most efficient use of the trucks on the road.

The trick is to keep this tables synchronized across all the clients without having the clients poll the database every x seconds to see if there are any changes. Such strategy simple does not scale. What you do in this case is you set up a messaging server in the back end like JBoss Messaging or TIBCO, create a topic in the Messaging server that all the clients subscribe to and then every change made to the table by clients, or maybe a GPS service, is published to the topic and all clients get a new message and update their table. This is a more efficient model.

How do you achieve a similar push architecture with AJAX? It is easy to create a dynamic AJAX table that updates itself asynchronously without ever resubmitting the page. You can do it with straight Javascript and HTML, or you can use a library like Google's GWT or DWR. The problem is that the AJAX page will eat up your server resources. By constantly polling for changes, you will have constant queries on your database when they are not necessary. But, how do you implement messaging with AJAX?

If you want to make your life easier, use Apache ActiveMQ as the JMS provider.
ActiveMQ provides Javascript remoting objects for the AJAX pages to listen for messages asynchronously. If you want to use another framework or JMS provider, you will need to create a POJO or servlet on the server side that writes and listens to the JMS topic, and the remoting Javascript objects need to listen to events from that POJO or servlet. JBoss is working on a similar feature for their JMS product. TIBCO also has AJAX libraries and support.

With all the messaging support for messaging and AJAX in the market, there is no excuse to use Pull Architectures in AJAX clients anymore.

Email me if you need more details.