Monday 8 December 2014

Static or dynamic types?

Last friday, Gilles and I have organized the second edition of the "Mate, you have to see this!" event.

Gilles Philippart is the head of the tech-coach unit within our company (30 passionates developer-coaches that are transversally sharing their skills & experience all over the organization). He is also a big, BIG, fan of the Clojure language (and co-organizer of the Paris Clojure User Group).

"Mate, you have to see this!" is a kind of event that I was recently launching within our organization. The concept (already described here) is pretty simple:

  1. We book a meeting room at noon
  2. We project and eat our sandwiches in front of an inspiring talk in video that one of us has really liked
  3. And we all debate afterward (it's a 2 hours event)

For the first edition last month, I had chosen an excellent french talk from Gregory WEINBACH (related to DDD).

This time, I've setup a doodle poll and we had reached the quorum for Rich Hickey's "Simple made easy" talk. At the end of the day, some of us recommended to replace it with a more seminal talk from Rich Hickey: Are we there yet?. I was not convinced by this choice actually (I find "Simple made easy" more inspiring & accessible), but we ended with it for our session.


Did I told you that Gilles was very pushy around Clojure adoption? ;-) Yeah? no?

Because this time, he had contacted two major french ambassadors of Clojure to join us for watching the video and debating afterward. Those french ambassadors are:

And to be honest, even if I really enjoy Rich Hickey's talks in general, the Q&A session at the end was far more interesting for me.

Let me sum up some of them for you:

The language

"Why did you choose Clojure and not another more-or-less functional language (like: Haskell, F#, ERLANG, Scala)?". 
Christophe GRAND -that started to use Clojure just few months after its creation- explained that he wanted to try something else at the time (than Java) and that the vision & the mindset of its creator was decisive. Rich Hickey is a pragmatist, and his programming language is part of a new breed of functional programming (FP) languages that are very pragmatic too (like F#). With their interoperability capabilities, we can use them in our day to day job (vs. Haskell).

Hiram explained that he wanted to exit his confort-zone and to find an alternative to Java few years from now. He was so amazed by Clojure (& its expressiveness) that he's now trying to favor as much as he can Clojure for the applications he's working on.

Clojure's toolset

"Compared to a mainstream languages like java or C#, how is the Clojure development environment today (IDE, test tools, etc)?" 
Well... it seems that all the reasons that made LISP a painful experience for some of us (remember your 'vi' sessions without any help nor syntax highlighting?) are obsolete with Clojure & its relatively mature ecosystem.

Code-Designing Process

"FP removing the side-effects that we found within OO code, do you still need to write tests? If yes, when do you write them? And is TDD a valid approach with Clojure/FP?"
Ok, neither Christophe nor Hiram are following TDD to drive their design process, but both of them confirmed that they write tests for their Clojure applications. In fact, REPL -Read-eval-print loop- is the alternative here (i.e. the ability to interactively execute part -or- all the lines of code from within a console). The usage of the REPL facilitates exploratory programming and debugging, for something that Uncle Bob is calling "Eye Driven Development" (EDD). The developer in that case is "playing" with his code within the REPL until he's reaching what he wants. Once he has found a satisfying result, he can then put his code under a test harness (for non-regression, but also for a means of documentation). But even if both Christophe and Hiram don't do TDD in Clojure, it seems that other guys do (like here, or like Jérémie GRODZISKI -according to my mate Radwane HASSEN).

For me who am not strongly confortable with a FP language yet (lazy bast... I am), this is a very intriguing question. I mean, I can easily grasp the utility of the REPL for building and troubleshooting small functions. When I TDD with tools such as NCrunch and NFluent (in .NET), I somehow transform my augmented IDE into a kind of REPL (to have this immediate feedback on my baby steps). But I can barely grasp how to leverage on the transiant REPL for more than few small functions. For bigger code bases I mean. Will I pollute my developer's head with tons of contextual details (such as: did I include all the relevant code parts for this experimentation to be relevant)? But for now, I guess I need to learn and play more with FP languages than I already done (in a casual mode so far).


"Since Clojure is hosted within a JVM, are you sometimes jeopardized by stop-of-the-world-GCs?" 
As a disclaimer I would add here that every time Rich Hickey is talking about "good performances" within his talks, you can translate it into "good throughputs". It's a fact that Clojure code - with all those immutable data structures - is made for parallelization. On the other hand, an important memory allocation policy under GC may have severe impact on your application latency.

Well... even if Clojure immutability usually embraces short-lived memory allocation -which is very GC friendly- Christophe admitted that for some "niche" usages, he has been forced to tweak the JVM, to hunt down hotspots and even to rewrite or optimize part of his corresponding Clojure code in rare cases, to reduce allocation. Fortunately this is not needed for all Clojure applications, but only for the very stressful ones like we can found sometimes in financial front-office (i.e. a niche usage).

We also shortly talked about the concept of Software Transactional Memory (STM) that Rich Hickey addresses in his "Are we there yet?" (not the best part of his talk according to me, and a clumsy topic actually).

But our conclusion for this performance "chapter", was that we need to measure, to measure everything before going into production (confronting our systems to our use cases and contraints; not to our intuitions ;-). In clear, that means performance tests for everyone ;-P


"Do you have an example of a real Clojure application that is under production since various years and that has been maintained by someone else than you?" 
Was one of my questions. Haunted since many years now by this maintainability concern (i.e.: what makes a system surviving to changes? how to preserve original intentions despite turn-over?), I am curious about this challenge with FP languages. Will the utter concision of some FP language make this challenge even more complicated than with OO mainstream languages? As much as complicated? Christophe told us that he had an example of that for one program he started to build for the CERN various years from now and that has been upgraded and released by other guys once Christophe had left the site.

Static or dynamic types?

As I was explaining to the guys that I was currently hesitating between starting to learn F# or Clojure, the major difference between those 2 pragmatic FP languages aroused the discussion.
Before we had this discussion, my gut feeling favored the static type option (thus like F#), but
according to Christophe, it seems that there is a interest for not relying too soon on strong types when you are designing your application. Indeed when you start a program, there are lots of undefined things, including the kind of data you will work on (usually coming from outside/integration constraints). Creating types to early may be painful, but also hurts the reuse of your applicable functions and limits your expressiveness.

Radwane, currently looking at F# seriously was saying that he really enjoyed its static type system. According to him, it's so easy and fast to declare a type in F# (one or 2 lines) that it wasn't really a problem to create them. As a side-effect ;-) : "when your program builds, it means it works".

Well... I guess I'll have to play a while with those 2 languages before I can make my choice.

Anyway. I'd like to thanks here again Christophe and Hiram for having join us and made this "Mate, you have to see this!" episode so interesting.

Wednesday 3 December 2014

Will the Ultra Messaging guys wake up AMQP?

Originated in 2003 by John O'Hara at JPMorgan Chase in London, the Advanced Message Queuing Protocol - which aims to be the wire-level standard specification for messaging interoperability (vs. API-level standardization like JMS) - is catching on rather slowly in the contemporary landscape.

To be honest, before yesterday I was very skeptical about the future of AMQP as a standard. Now, I'm still not enthusiastic, but a little bit less skeptical ;-) Reasons why:

With a version 1.0 less than more adopted by the various broker-based projects (see here for details), the new things for AMQP would probably come through a broker-less (and daemon-less) product: Informatica Ultra Messaging (a.k.a. IUM & formerly known as 29West).

Why a broker-less & daemon-less solution decided to implement AMQP?

Started few years from now as the fastest "low latency" messaging solution for finance, Ultra Messaging is now covering all the standard use cases of Message Oriented Messaging (MOM) year after year. Adding persistence, HA-support, load balancing (with stickiness!) and even once and once only guaranteed delivery.

From a "nothing in the middle" overall design approach allowing us to pick the transport we need on a per-use case basis (i.e. Reliable Multicast, Reliable Unicast, TCP, RDMA, IPC, SMX, ...), Ultra Messaging was familiar with the concept of adding components aside their peer-to-peers architecture. Not in the middle like we have in a broker-based architecture. A-side the critical path of the data distribution.

A full AMQP 1.0 solution

With its upcoming version 6.8 that will be release December 10th 2014, IUM will now support both a new AMQP transport, and a fully compliant AMQP 1.0 broker. That won't change the core & existing transports of Ultra Messaging, but simply provide a new feature (AMQP) if needed.

In fact, Ultra Messaging should deliver the first AMQP broker-based solution supporting High Availability (HA) with full automated fail-over mechanism.

No NIH syndrome

Initially started to work on their own queue-based architecture & component a few years from now (i.e.: UMQ), the Ultra Messaging guys were asked by their customers to support interoperability scenarii with existing market or open source solutions. You know, the kind of solutions already deployed for the guaranteed delivery use cases: TIBCO EMS, Apache ActiveMQ, IBM WebSphereMQ, Pivotal RabbitMQ, etc.

Instead of continuing to implement their own UMQ solution, the Informatica guys wisely decided to leverage on an existing one: Apache ActiveMQ, and to improve it. In fact they've forked it in order to provide the product expected by their customer (including HA). To do so, they have added a part that is currently missing in the AMQP space & standard: the broker-to-broker interactions.

At the end of the day, it's a nice occasion for the Ultra Messaging team:

  1. To provide their customers a full HA compliant AMQP broker (relatively mature) solution
  2. To provide their customers a new AMQP transport within the IUM toolset,allowing interoperability scenarii with other AMQP 1.0 brokers (for AMQP idealistic people?)
  3. To contribute back to the Apache ActiveMQ project (with their pull requests)
  4. To wake up the AMQP standardization initiative by suggesting something for the broker-to-broker interactions part

The truth about AMQP...

I would like to end this post with an anecdote. While I was in React Conf last April in London thanks to my Adaptive friends, I had the opportunity to dine with the speakers. During the last night, I was located right next to Peter Hintjens, creator of  ZeroMQ.

I had a few drinks that night (London, you know... ;-) but I can clearly remember Peter saying to me that AMQP was originally his idea... but also the trigger for him few months after, to create ZeroMQ (when he saw that AMQP was leaded into a political nest). Fortunately, I recently found this old post confirming that I wasn't drunk when Peter was explaining that to me. Extract:

"the AMQP spec which Red Hat read in 2007 was largely my invention: I dreamed up exchanges and bindings, hammered them into shape with my team and the guys at JPMorganChase, explained how they should work, explained how to extend AMQP with custom exchanges, wrote thousands of pages of design notes, RFCs, and diagrams that finally condensed — by my hand and over three years — into the AMQP spec that Red Hat read in 2007."

I never seen any mention to Peter's contribution within the AMQP site or standard, but I thought it was important here to Render unto Caesar the things that are Caesar's.