Monthly Archives: December 2008

Coherence across a WAN? Push Replication rocks!

We’ve done a lot of work in the past six months to simplify how to use and deploy Coherence Data Grids around the globe.  It’s always been possible to do these things – Coherence provides some great infrastructure like *Extend to do this – but we’ve lacked a concrete framework.  The Push Replication Pattern is making some serious in-roads to solve some of the challenges faced when designing a globally distributed Data Grid.

There are some great advantages of this pattern;

  1. We’ve provided the complete source code for it.  You can embed, change, enhance it as you like.  No restrictions.
  2. It provides support for completely asynchronous, but guaranteed in-order updates (with batching) between multiple-sites (Data Grids).
  3. It provides a completely pluggable infrastructure layer to programmatically resolve data conflicts between sites.
  4. It avoids the classic problems of other approaches, like explicitly setting up dedicated and single point of failures with Mirror Services, Gateways or routers.  The solution simply embeds in your application.
  5. It’s completely monitorable via JMX.
  6. It supports almost every type of WAN replication/synchronization scenario, including; one-to-one (uni-directional), one-to-one (multi-directional), many-to-one (uni-directional “centralized”), one-to-many (uni-directional “hub”), many-to-many (multi-directional “mesh”) architectures – with all of the above mentioned guarantees.
  7. It’s not just a theoretical pattern – it’s in production in several large projects.
  8. We’re constantly enhancing it based on customer demand and feedback.

I’ll leave you with this quote:

“The [push replication] tools are working very well.  This is very good news for us, as it allows us populate and re-populate caches across the globe in a safe and consistent way.  Thanks…”

Coherence .NET: Just Works…

It’s been a while since I’ve had a chance to play around with Coherence.NET (mainly because I predominantly been working with Java). However over the weekend I decided it was time to put a .NET and Excel front-end on one of my Coherence demos.  

I essentially have an application that keeps a few hundred thousand financial positions in cache (call it a Data Grid if you like) and would like to run some queries against them from .NET and later Excel.  Even without .NET equivalent classes of my Java object model, it was cool that I could still run a lot of queries out-of-the-box.

eg 1: Getting an .NET INamedCache – the equivalent of a Java NamedCache – was easy;

      INamedCache positionsCache = CacheFactory.GetCache("positions");

eg 2: Determining how many positions I had, also easy;

      int nrPositions = positionsCache.Count();

eg 3: Determining the distinct symbols for the positions (an aggregation);

      Object[] result = (Object[]) positionsCache.Aggregate(AlwaysFilter.Instance, new DistinctValues("getSymbol"));

The AlwaysFilter.Instance specifies that all positions must be considered.  While it’s possible to specify any other filter, in my example I’m looking for distinct symbols across all positions.  The result of the aggregator is an Object[], simply because the result of the DistinctValues aggregator is an Object[].  Note however that the Coherence permits aggregating any type of value (especially non-primitives).  In this case we’re aggregating the result of the “getSymbol” method on the positions class.  This aggregation is performed in parallel across the Data Grid.

More examples in the next few days.

OS X Leopard: Extending Quick Look to support XML

Personally I find Apple’s Quick Look feature in Leopard is a massive time saver. The problem however is that out-of-the-box only a few file types are supported – you know the usual documents like photos, pdfs etc.  What I really wanted was the ability to take a quick look at an xml document – rather than dragging them into TextEdit.  The solution was pretty simple.

As usual Apple has done a great job of making Quick Look pluggable.  I was pleasantly surprised to find a Quick Look plugin site (called QuickLook Plugins) that has a bunch of useful extensions, including one for XML.


 Job done.  Now I can quick view almost anything.

Coherence: Customer References and Organizational Scalability

One of things I think we really could have done better at Tangosol was to tell more people how successful we and Coherence really was (and continues to be).  It’s something we always wanted to do, but to be brutally honest, we either a). didn’t have the time – our first priority was and continues to be ensuring success rather than promoting it, and/or b). customers didn’t want to disclose their architectural internals, including their use of Coherence – which I think is completely reasonable as their use of Coherence often yielded significant competitive advantage (especially in trading systems). Don’t get me wrong.  We had and continue to collect some amazing customer references, but at the time we just didn’t promote them or ourselves as well as I believe we could have – Cameron Purdy (one of the founders of Tangosol) would probably agree.  

Of course, this is now starting to change as being part of an organization the size of Oracle, people are confident we’re going to be around tomorrow – there is simply no doubt!  Better still, Oracle has global teams that explicitly focus on working with partners and customers to both promote their use of technology and their adoption of solutions like Oracle Coherence.

While I can’t comment on the number of customers or projects that currently use Oracle Coherence (or about it’s growth rates), figures available at the time of the Tangosol acquisition by Oracle suggested that the closest (in terms of supported deployments) of a commercial / open source product in the same market place (say in-memory clustered caching etc) was less than 10% of the number of deployments of what was then Tangosol Coherence.  In other words, from what we could gather, the number of Tangosol Coherence deployments was close to 5x of all of the combined solutions in the same space.  Cameron has always said Coherence was the market leader, and due-diligence at the time clearly demonstrated that (and it continues to be by a long way).

At first I simply didn’t believe this.  I couldn’t.  How could such a small organization deliver so much to so many corporations?  How could such a small organization even support such a large number of deployments?  How could an organization support the types of mission-critical applications running on Coherence.  This should not be possible.  

But after some thought the recipe for success became clear.

1. We don’t build products and then try to find a customer.  
All features in Coherence, in fact Coherence itself, originally came from customers. Coherence is probably one of the only solutions in the Data Grid space that started like this.  We started with a paying customer.  We didn’t think up a cool technology or take an existing technology and try to build a company/new market around that.  Tangosol was 100% employee owned and 100% profitable, with compounding growth every year in operation.

2. We ensure customer success by working with customers.  
We know if customers fail, we fail.  Apart from producing a solution this means quality, precision, stability and consistency must be ridiculously high.

3. Customers needed to value what they did.  
If customers could not financially value what they did, it was likely they couldn’t financially value our help or Coherence.

4. Customer failure was not a option.

This does not mean we’d do or do anything for anyone.  Nor does it mean we’ve always been successful.  Quite contrary.  If we felt something like Coherence was (or is) a bad fit, either strategically, architecturally or organizationally, we’ll happily suggest to use another approach or solution, including recommending a competitor product (which of course annoys the hell out of our sales people!).  Our goal, and certainly mine personally, was (and continues to be) to ensure customer success.  Hopefully Coherence could or will play a part in that success, but if it didn’t then we/I weren’t offended – still aren’t.

So where are we now?  Well contrary to the usual competitor propaganda and FUD (probably the most annoying thing about this market place), Coherence global adoption is simply amazing.  No it’s not being killed off (as was suggested when the acquisition was announced).  No we’re not forcing customers to purchase other products (as was also suggested).  It’s not being shelved as a “cash generator”, and we’ve not stopped developing and all retired (as was also suggested).  It’s powering ahead.  The original engineering team still continues to rock and churn out amazing stuff – though the team now is much bigger!

No this is not marketing or self-promotion bullshit.  

Think of it this way;  Tangosol in April 2007 had around 10 sales people – globally.  I think Oracle had closer to 15,000 (even before BEA joined). Oracle also had in excess of 300,000 existing customers (according to the Open World banners).  So in terms of scaling the Tangosol organization, we’ve essentially had a 1000x+ increase sales force alone – in just one year.

Even with a completely pessimistic view of Coherence, the potential adoption, even just internally, would dwarf the size of the business prior to the acquisition.  My guess is that no other product in the same space as been given the exposure or level of investment.  

To say we’ve been busy is an understatement.  We’ve had to invest a lot in training a huge sales organization, a global support team, consultants, the training organization, pre-sales and engineering (yes, the Coherence engineering team has also grown significantly).  I personally was simply amazed at the size, scale and agility of operations at Oracle (more about this in a later post).

It’s been a massive 18 months and I’m certainly looking forward to the next, especially the forth coming release of Coherence 3.5, that of which Cameron talked a bit about this week in London.