Debunking the stupid myth that performance is a technical concern

I often hear people saying that latencies and response times are not business topics. I strongly disagree with that vision, and I like the punch-line used by Gojko here: “Debunking the stupid myth that performance is a technical concern”.

Indeed, It’s a fact that extra latencies or bad response times strongly impact the business. As a customer, I can’t stand to wait ages (even minutes;-) until someone (a salesman, a web site,…) is able to answer my questions, or to help me to quickly buy and checkout the product I already identified myself. It’s rather more true nowadays where our expectation level has been dramatically increased with our mobile phone usages.

Asking our business questions like : “what are your objective in term of performance?” is rarely productive. With such approach, we usually end up with average values for our implementation. Clearly the kind of setup fully loaded of implicit, leading to unclear situations,inappropriate architecture choices, and often crisis and strong tensions when we experience fires under production. Sad panda ;-(

On the other hand, by asking our business questions such as “ok, you want an average response time of 1 second, but is it ok for some things to take more than 10 minutes once a day?”, we can start to obtain reactions, deeper involvement, and answers that will  help us to build and validate the service level expectation we need to leverage on.

By leveraging on, I mean: to be verified via a continuous performance test harness, and carefully monitored under production (capacity management). A much more mature way of supporting the business of our clients.

I’d like to end this post with a reference to the excellent presentation of Gil TENE: “How NOT to measure latency”. In particular, Gil is sharing with us the typical kind of questions he asks to his clients in order to establish the performance requirements/service level expectations. 

The outcome of such exercise is something like:

• 50% better than 20msec
• 90% better than 50msec
• 99.9% better than 500msec
• 100% better than 2 seconds

The entire presentation worth the look, but the typical questions for the interview are showed from the slide #98.

Very useful…

Simple Binary Encoding (SBE): the next protocol buffer?

Even if it's still in beta, the SBE ultra-fast marshalling API (available in C++, Java & .NET) seems very promising. Designed from a financial (FIX Trading community) standard, this first implementation is fully open source, and made by -wait for it ...

• Martin Thompson (former LMAX CTO) 
• Todd Montgomery (former 29West CTO)
• and Olivier Deheurles (for the .NET part)

If you want to discover both the big picture, the history, and how you can use SBE, check the nice post written by my friend Olivier. 

Another really interesting part to read is the description of the SBE design principles chosen by the authors and published on the SBE wiki. A clearly mechanical sympathy approach ;-)

No doubt this has to be closely followed...

Metrics, Metrics everywhere...

... is a truly awesome presentation by Coda Hale. Even if its content is really interesting (in a nutshell: improve your decisions and time to market by measuring your code when it runs), I really like the format of his presentation. 

Very funny, fluent, and built like an Anaphora (repetition, repetition, repetition), his presentation is a real 'model of genre' in term of efficiency to spread a message. 

Available on youtube, Coda's presentation lasts only 30 minutes long (there are 15 minutes of Q & A at the end); it definitely worth the look:

http://www.youtube.com/watch?v=czes-oa0yik

'As developers we have a mental model of what our code does ... we spend so much time inside our heads, it's very easy to mistake what's inside of our heads for reality (i.e. to mistake the map for the territory)... we can't know until we measure it'

+1

(thanks to my mate Cyrille for this resource)

The technology behind an equity trade

I highly recommend you to watch the excellent presentation of 'The technology behind an equity trade' made by John O'Hara during last QCon London 2013.

Clear as crystal and very informative, John is giving a good introduction of main bank functional blocks, before explaining the challenges every bank will have to face in the upcoming years (starting now ;-)

Finally, he ends explaining why technology will be so crucial in that industrial (and digital) transformation. Very nice session!

The reactive manifesto

Working in financial front-office since several years from now, I'm used to build reactive systems. 

In a nutshell, they are systems that are ready to respond to stimuli (whether market data updates, client requests for prices, spreads modifications, position changes, etc), in order to react as quick as possible (in milliseconds or less), in a world where you must constantly protect you against waves or throughput peaks of data coming into your systems.

To achieve that, we mostly build systems that are event-driven, stream-oriented, scalable (vertically, but also horizontally), performant (with conflation enabled, GC-less strategies, ...), and resilient (i.e. recovery-oriented & production ready).

Today, one of my mate showed me this new reactive manifesto which is all about that kind of systems we used to build. I found it very readable and informative. Check this out by yourself:

http://www.reactivemanifesto.org 

And welcome to our reactive world!

Low latency framework: the must read presentation of Richard Croucher

I've just read a very informative document written by Richard Croucher about how to build low latency systems. One of the best paper I've ever read on my favorite topic (very synthetic and clear).

It's available here.

About Rx performances

James Miles recently shared some performance figures and explanations related to the latest Rx Performance Improvements.

With this latest v1.0 stable release, Rx seems to enter in a new era. It's a very good news for .NET developers...

Write performant code: keep some fundamental figures in mind

It's funny to see the amount of time spent (i would say lost) by some developers to optimize their code in some location... where the ROI will be peanuts at the end of the day ;-(

Worse: premature optimizations. The real ones, I mean (I heard you Joe, and I agree ;-). The ones where  peoples sacrify readability and evolutivity to the hostel of performance (really? did you measured it concretely?). In most cases, a simple but wise choice of relevant types and datastructures within our code save us lots of time and energy without creating maintenability nightmares.

If you want to develop low latency and scalable solutions, it is obvious that you should know the core mechanisms of your platform (.Net, Java, C++, windows, linux, but also processors,  RAM, network stacks and adapters...). How the GC works, how the memory and proc cache lines are synchronized, etc.

But do you have in mind the cost of some elementary operations (in term of time and CPU cycles) when you are coding? How long it takes to make a classic network hop? How long it takes to make a typical I/O read? How long to access the memory depending on its current state?

As a reminder, here is some figures (some are borrowed to Joe Duffy's blog) that you should definitely post-it in front of your development desktop. If you don't want to improve your code blindly, it's important to know what things cost.

• a register read/write (nanoseconds, single-digit cycles)
• a cache hit (nanoseconds, tens of cycles)
• a cache miss to main memory (nanoseconds, hundreds of cycles)
• a disk access including page faults (micro- or milliseconds, millions of cycles)
• a local network hop with kernel-bypassing RDMA & 10GigEth (sub 10 microseconds)
• a LAN network hop (100-500 microseconds)
• a WAN network roundtrip (milliseconds or seconds, many millions of cycles)

The ultimate MOM?

I've never had faith in silver bullets ;-) but I have to admit that Solace (appliance-based) solution is very attractive for a pre-trade (but also post-trade) financial entreprise messaging system.

Pros

Because Solace use silicon instead of software for their "hub and spoke" oriented messaging solution (fully compliant with JMS standard, but with much more features) , there is no OS interrupts, context switching nor data copies between kernel and user space for the "hub" part.

I still didn't had the chance to evaluate it, but on the paper and according to some of my ex-colleagues that had made evaluations,  Solace looks like a kind of ultimate solution for financial entreprise messaging system.

"Reliable delivery with average latency of 22 microseconds at 1M msgs/sec", "Guaranteed delivery with average latency of 98 microseconds at 150,000 msgs/sec",  "10 million topics with support for multi-level, wild carded topics", "9,000 client connections"... (http://www.solacesystems.com/docs/solace_corporate-intro.pdf). Such figures make me dream and remind us that software can't win over hardware for those message-oriented middleware (MOM) use cases...

Their several white papers are very relevants and informatives. In particular, the one that explain how to build a single dealer platform 

(meaning: a web oriented application produced by an investment banking in order to allow all its clients to directly cope/deal with it). Oh Yes, because the future version of their appliance will also allow to bridge with http clients (one more killer feature ;-)

This particular white paper is available from here: http://www.solacesystems.com/solutions/financial-services/single-dealer-platform

Cons
More than the (high) price of such a solution, and even if it is increasingly used in some banks, perhaps risks regarding the sustainability of such hardware-based solutions that has to be lifted.

Indeed, what would happen if their (unique?) appliance production factory get burnt? how long would it take to Solace to replace this mass production to fulfill contracts, etc.?

Low latency systems in .NET

Not new, but since i'm currently in a low latency mood ;-)

Back to june 2009, Rapid Addition, one of the leading supplier of front office messaging components to the global financial services industry, published a white paper with Microsoft on how to build ultra-low latency FIX engine with 3.5 .NET framework...

Their whitepaper (and RA Generation Zero framework) mostly indicates how to prevent from having gen2 garbage collection.

I really like their description of a managed low latency system with a startup phase (with memory resource pooling preallocation), a forced full GC phase, then a continuous operation phase for our low latency systems (where we avoid to cause garbage collection by using and recycling resource pools).

According to me, this white paper demonstrates that Rapid Addition really know what they are talking about.

In another press communication, they have said that their FIX engine breaks the 10 micro seconds average latency with a 12,000 messages/second throughput (still with the 3.5 .NET framework based solution).

Very informative.

A new perspective for Ultra low latency performant systems

I just watched an awesome presentation by Martin Thompson and Michael Barker. They explained how they implemented their ultra low latency (with high throughput) systems for the London Multi-Asset eXchange (LMAX) and it's pretty impressive: 100 000 transactions per seconds at less than 1 millisecond latency in Java...

Since I'm working on FX and low latency systems in general about several years from now, I was very interested by their teasing (100K...). I have to admit that I was thrilled by their presentation.

For those that didn't have one hour to kill watching their video, here is a summary:

---HOW TO DO 100K TPS AT LESS THAN 1ms LATENCY----------------------------

1. UNDERSTAND YOUR PLATFORM
2. CHECK YOUR PERFORMANCE FROM THE BEGINNING
3. FOLLOW THE TIPS

---------------------------------------------------------------------------------------------------------------

UNDERSTAND YOUR PLATFORM

• You have to know how modern hardwares work in order to build ultra low latency systems
• The advent of multi-cores with their bigger and smarter caches (do you really know about how proc cache synchronization is working? false sharing drawbacks? etc)
• Ok, free lunch is over (for Ghz), but it's time to order and use more memory!!! (144GB servers with 64bits addressing for instance)
• Disk is the new tape! (fast for sequential access); rather use SSDs for random threaded access
• Network is not slow anymore: 10GigE is now a commodity and you can have sub 10 microseconds for local hop with kernel-bypassing RDMA
• (Not hardware, but) understand how GC and JIT work (under the hood)

CHECK YOUR PERFORMANCE FROM THE BEGINNING

• Write Performance tests first
• Make it run automatically and nightly to detect when you should start to optimize your code
• Still no need for early and premature performance optimizations

FOLLOW THE TIPS

• Keep the working set in-memory (data and behaviour co-located)
• Write cache (cache-lines synchronization) friendly code (the rebirth of the arrays ;-)
• Choose your data structure wisely
• Queues are awful for concurrency access, rather use Ring Buffers instead (no stress: we just said that we bought lot of memory ;-)
• Use custom cache friendly collections
• Write simple, clean & compact code (the JIT always do better with simpler code-shorter methods are easy to inline)
• Invest in modeling your domain. Also respect the single responsibility principle (one class one thing, one method one thing,...) and the separation of concerns
• Take the right approach to the concurrency. Concurrent programming is about 2 things: mutual exclusion and visibility of changes which can be implemented following two main approaches. i) a difficult locking approach (with context switch to kernel), and ii) a VERY difficult atomic with no blocking (user space) instructions (remember how are implemented optimistic locking mechanism within databases). You should definitely choose the second one
• Keep the GC under control. Because the GC may pause your application, you should avoid it by implementing (circular buffer) preallocation and by using a huge amount of (64bits) memory
• Run business logic on a single thread and push the concurrency in the infrastructure. Because trying to put concurrency within the business model is far too hard and easy to get wrong. You would also turn the OO programmers dream of: easy to write testable and readable code. As a consequence, it should increase your time to market.
• Follow the disruptor pattern which is a system pattern that tries to avoid contention wherever possible (even with business logic running on a single thread)

---------------------------------------------------------------------------------------------------------------

Ok, this presentation leave us with lot of questions about how they implemented their systems. However, I found this video very refreshing and interresting regarding several points (the need for huge amount of RAM with 64 bits addressing, the interest of network kernel-passing technologies with RDMA, the fact that queues are bad to handle concurrency properly, that clean and simple code doesn't prevent you from having excellent performances, that concurrency should stay out of the business logic, that the separation of concerns allow you to have very competitive time-to-market,... and of course that you need a write-performance-tests-first approach).


For the disruptor pattern explanations and much more, see the entire video of this presentation here: http://www.infoq.com/presentations/LMAX

Note: don't see this video in full-screen mode otherwise you will loose the benefits of the slides displayed below ;-(

Cheers, and happy new year to everyone ;-)