Over the past few years I’ve seen several financial system projects go horribly wrong (well almost), not through any fault of the architects and engineers, but due to what I like to now call an unfortunate “Crying Game” experience.
If you haven’t seen the movie “The Crying Game” and would like to enjoy it without me giving the twist away, stop reading now! If you’re not going to see the movie, you can get a brief synopsis here. If you’ve seen it, enjoy the following while thinking about the twist in at the end of the movie.
In essence a “Crying Game” experience occurs when you’ve made an assumption about the “fitness of purpose” of a technology component, either commercial or open-source and you discover just as you’re about to “go live” (or bet your business on it) that it won’t perform, scale or behave as you expected, regardless of what you do to it, how much source-code you’ve got or what other people and vendors have lead you to believe.
You’ve invested a bunch of time, effort, energy and for that matter passion into a technology only to unexpectedly discover that what you trusted is now completely unacceptable to you. It’s worked perfectly in all of the situations you’ve investigated, you’re very happy with you’re choice, but the “making it happen for real” experience goes horribly wrong due to an “unexpected surprise”.
We’re not talking about a “surprise” like the logging format isn’t quite what you wanted. We’re not talking about a technology that’s been working for a while and is slowly failing some SLA. We’re talking about new information that you discover for yourself as you attempt to go live or early in production that breaks all of your previous assumptions, your architecture and ultimately risks your business.
Like in the movie you’re left with very few choices;
a). Abandon it. Wash your hands of the situation. Walk away and start again, potentially not telling anyone about your potentially embarrassing experience. Obviously this is not often an option in the technology business. Architects that do this soon become well known – all for the wrong reasons.
b). Accept the situation and attempt to work with what you’ve got. You’ll never be truly comfortable and will inevitably feel you’ve been ripped off or let down. If you’ve using a commercial solution you can obviously work with the vendor to “resolve” the issues. If you’re using an open-source solution you can “fix it yourself” or “contribute a solution to better the community”. Alternatively you can get “help” from someone experienced in such matters. Ultimately “surgery” is being proposed. And as we all know, last minute surgery, especially DIY, is going to be messy. You’ll end up with a mutant.
c). Get angry, argue with the solution provider and then replace it with something more appealing – possibly telling everyone about your experience.
d). Change your business to match how the solution provider believes you should work. Again, not usually an option.
Regardless of the decision you take disappointment may reign for a long time. “You’ll never do that again” right?
Crying Games seem to occur for the following reasons;
1). Lack of adequate testing / domain knowledge / technical experience.
While it’s easy to say “you didn’t test enough”, there is only some much testing you can do. There comes a point when you simply have to trust the information you have (from your tests) and trust your expert judgment.
Given the Crying Games I’ve seen I would never claim that the architects, engineers and projects failed in their duty to test. Far from it. I would say they tested more than adequately and decisions were soundly based on the information they had at their disposal.
2). Lack of adequate references.
Taking references is always important to avoid unexpected and costly surprises. Unfortunately “references” these days are always “positive” right? It’s very hard to find referees that offer full disclosure, especially if there is the thought that they may be flamed, harassed, bullied or sued by a vendor / community. The worse kind of referee is one that has been paid-off. ie: gets a discount on their adoption to offer references in the future for others.
If you can find a previous partner that will give you a truthful reference you’ll have a much greater chance of avoiding a Crying Game with your new architecture!
3). Lack of adequate interrogation.
Sounds nasty, but there is nothing like asking tough questions of solution providers. eg: What was your worst loss? Why? What was your best win? Why? What is the worst part of the product? Why? What is the best part? Why? When shouldn’t you use this product? Why? Who are your competitors? Why are they competitors? What deals have you lost to them? Why? Can I talk to them? Which deals did you win from them? Why? What would be a disaster to your business? What is your plan to avoid this? Who do you depend on? Why? Can I talk to them? What relationships / components do you depend on? Why? Have you done this before? When? With whom?
I think you get the picture. Attempt to find out what they are like “without all of the marketing makeup”. Interrogation reveals a lot of information. Do it fast. Be up front. Don’t drag it out. If you can, and you might not be able too, ask the same questions of the provided referees.
4). Lack of integrity.
There are basically two forms of deception; deliberate and accidental. While both are very worrying, being deliberately mislead during the process of adopting software or hardware for an architecture is less likely to be forgiven, especially if your business (or employment) are at risk.
Given the recent projects on which I’ve been involved, it seems that beyond everything else integrity is key. If you can push technology egos to the side, ensure that marketing and sales objectives are not driving factors in your decisions, you’ll probably avoid a Crying Game.
If you ever have concerns about technology providers or your decisions, let people know. It’s not a sign of weakness, but strength. It’s not doubt, but a chance to test early. Always try to break what you have. Break your own ideas. Break others. Fail Fast.
Failing slow seems to put architectures on a path to a Crying Game.