Entreprise Architecture, TOGAF, and the ol' pitfall of big upfront design...

Executive summary:

Entreprise architecture is answering to real concerns & challenges for big-scale companies. After just being TOGAF certified, I don't think that this approach "applied by the book" is the proper answer to support those challenges for any situation where software development is concerned.  Reasons why detailed here (note: allow 27 hours of reading :-)

----------

This week, I followed a training course validated by 2 exams at the end (theory & practice) in order to be TOGAF (9.1) certified. This course, now mandatory for almost every architect in my company, was about Entreprise Architecture (TOGAF stands for The Open Group Architecture Framework).

Wait a minute! What entreprise architecture stands for?

Good question! and as a DDD practitioner, I think it worth here to define our ubiquitous language for the rest of this post. Especially if you are an IT guy like me used to build and deal with softwares, you should not confuse the Entreprise Architect with what you think a Software/Technical Architect is. We are not talking about the same things here... Not at all. You should also not confuse Entreprise Architecture with IT Urbanism here (a kind of autistic & accountable-less french interpretation of Entreprise Architecture as explained by our TOGAF trainer ;-)  

Before defining Entreprise Architecture, TOGAF defines the Entreprise as "any collection of organizations that has a common set of goals". From a TOGAF perspective, an entreprise has a strategy and some capabilities to support this strategy (e.g. to sell its products in EMEA, to produce its products in China, ...). And capability is a keyword here. 

The Entreprise Architecture aims to guide the organizations to identify, specify and assess the changes necessary to execute their strategies. These changes are usually related to the management of their capabilities (did I told you before?). Whether to add a new capability (e.g. to be present on a new market), to change an existing one (e.g. to be compliant with a new regulatory constraint) or to dismantle an existing one (e.g. to stop the production of a given product). 

Of course since "software is eating the world" now, most of these changes will be supported by software development or integration, but having an Entreprise Architecture capability within a company is also important to support non-IT projects like: adding the "capability to build a production factory for our products in Italy in less than 2 years" for instance ... 

Ok. That makes sense to me. This being said, time to talk about TOGAF now.

The TOGAF vision

According to TOGAF, allowing the companies to execute their strategies in an efficient and safe manner is the job of highly skilled people that don't need to have IT knowledge: the Entreprise Architects.

All they have to do, is to leverage on technical experts from various domain to do their job (note from myself: if they think they need to do it...). 

What skills should have Entreprise Architects according to TOGAF? leadership, teamwork, oral and written communications, logical analysis, risk and stakeholders management. Lots of qualities, huh? simply too bad that TOGAF is not mentioning humility, empathy, and continuous learning / improvement mindset...

By the way, I explained the acronym but I didn't explained yet what TOGAF is. TOGAF is a framework including "the methods and tools for assisting in the acceptance, production, use, and maintenance of an entreprise architecture". 

There are 3 main pillars in TOGAF: 

1. The Architecture Development Method (ADM) which is a methodology cycle
2. The Architecture Content framework (describing the needed artefacts) 
3. The Architecture Capability framework (i.e. the description of what is needed to have an architecture capability within a company, including the governance aspects). 

But ADM is clearly the most important part of this huge document of 633 pages. ADM describes the 10 possible phases (steps and deliverables included) of every architecture project according to TOGAF.

ADM cycle and phases:

What I liked during this training

I must admit that I really enjoyed the introduction of this 5 days course, with statements coming from the (very smart) instructor like: 

• "Entreprise Architecture considers and supports the entreprise under the angle of its capabilities" (understand: use case driven)
• "the map is not the territory; we should always stay humble regarding any kind of modeling" 
• "TOGAF is an Utopia vision, it's mandatory to pragmatically bend this framework to our needs and entreprise's context"
• "Architects are not here to be simply conceptual: we are here to support people (i.e. our sponsors), and to demonstrate value. We need to deliver! We should not turn ourselves into IT urbanists (i.e. use-case-focusless ivory tower guys ;-)"
• "we should always know and remind us who we are working for (i.e. the sponsor of the entreprise architecture initiative)"
• TOGAF favors a "just enough" approach. If what you are doing is not linked to a requirement,  a risk mitigation or an impact. Stop working on it! ;-)
• The non-testable requirements are not acceptable
• "Using our veto card means that you've already lost as an architect"
• "There should be no architect within the entreprise architecture board! Indeed, the entreprise architects aren't the big persons (i.e. the executive decision takers with money). They are simply here to help big persons to take the good decisions regarding the objectives of the company (they've usually defined before)."

But it's important to notice here that most of those statements were coming from the instructor's experience, and not directly from TOGAF itself.

Regarding TOGAF now, I also appreciated that its ADM cycle was requirements-centric. ADM dedicates most of its phases to work on the "architecture" (from phase A to D); architecture meaning here "the description of the requirements to be taken into account". The "solution" arrives quite late in the process (mostly from phase E).

Since It's a common pitfall for everyone to be solution-driven instead of being use-case driven, I was quite pleased to find that mindset in TOGAF.

The 2 last thing I really appreciated within TOGAF, were:

1. The clear expression of the business/architecture principles (including their implications)
2. But also the exigences related to a good governance. Including the importance of the transparency of the architecture board decisions (minutes) to all the stakeholders (everyone within the company)

What bothers me within TOGAF? It's clearly not compliant with the agile and continuous delivery approaches...

Not really surprising, since TOGAF was initiated early 1990s. But even if TOGAF indicates that almost all phases of the architectural projects should be executed and reviewed iteratively, I fully disagree with its big upfront design approach. Indeed, TOGAF ADM cycle is basically saying: Entreprise Architects will specify and prepare everything BEFORE the implementation project is launched and transmitted to the involved PMO and dev teams. 

That doesn't match at all with the agile approach which is saying: confront your vision to the reality and gather feedback on your work as soon as possible... (remember: the map is not the territory...)

All the phases, steps and deliverables indicated within the TOGAF ADM cycle really reminded me the old Unified Process (UP). For those that are too young ;-) UP was the 1990s answer to the major tunnel effect of the former waterfall software methodologies. Ok, UP was far better than the waterfall-based processes, but its bureaucratic pitfall was one of the reasons why the agile manifesto was created at the time (too many deliverables per UP phases and iterations).

Also, the segregation made by TOGAF between the data & application concepts (see phase C) doesn't seem relevant to me. It reminds me the obsolete Merise french method ;-) According to me, considering the data aside the applications' use cases is the best way to fall into the pit of "mini-model the world" anti-pattern. i.e. the kind of situation where people are loosing their time on useless data/topics, simply to improve parts of the model that won't even have usage afterward (on that topic, see the excellent french presentation of Gregory Weinbach). 

Cause remember? A pragmatic architect understands that we should stop trying to model the real-world, but only what matter to our software usages instead.

There are also some good ideas within TOGAF (most of them are not new), but it clearly doesn't match with today's ways of building IT solutions (whether agile, lean, continuous delivery & lean start-up).

For non-IT needs on the other hand (e.g. "to build a factory from scratch in another country"), TOGAF seems quite applicable & interesting.

But the worst part for me is...

... the caricatural description of the actors involved that you can find within TOGAF (see the matrix presented in the chapter 52.5, or the extract below). Basically, it explains that the Entreprise Architects have all possible skills you can imagine. Even much more skills than the architecture board members (it's probably not the smartest communication move of the open group guys... Cause remember? the Architecture board members are the executive directors of the company that are sponsors of the entreprise architecture initiative ;-)

And finally, Entreprise Architects have much, much more accurate skills -of course- than the lame "IT designers" (solution architects? tech leads?). Hopefully for the Entreprise Architects' ratings, TOGAF doesn't retain "humility" as a criteria... ;-)

By the way, this lack of humility pitfall reminds me a former & excellent tweet from Martin Thompson (probably one of the finest technical architect you can find), saying:

How to be an architect:
1. Stop programming
2. Stop learning
3. Convince yourself nothing has changed since you did #1 and #2. 

;-) 

And finally, even if it's highly recommended to blend TOGAF and ADM to your context and organization, the huge number of outputs, tools and steps per phase (10 phases at max, with 8-15 steps per phases) can clearly lead us to a dogmatic and bureaucratic hell if the people in charge of setting up the architecture capability within our entreprises are not mature, or pragmatic enough.

Conclusion

More than TOGAF, the practice of entreprise architecture seems key to ensure that large scale organizations are able to align themselves to their strategy and to face the challenges related to the management of their capabilities.

TOGAF has some good ideas, but is intrinsically not adapted to agility best practices (nor continuous delivery for instance). Thus, we will need to find how to bend TOGAF strong enough to be continuous delivery compliant, or -more likely- to find something else than TOGAF for our modern entreprise architecture practice. 

But whatever the methodology and since the ivory tower syndrome is threaten every architect, it seems crucial to me to setup some risk mitigation/protection mechanisms within those organizations, in order to avoid situations where architect without continuous improvement mindset (and not aware that they need to seek assistance from other people's expertises) to be those who hold the keys to achieve the entreprise's objectives. 

When a developer stops learning things and improving himself as a professional, bad things happen. But when an architect is falling into the same laziness, VERY bad things happen ;-(

Wanna start the brainstorming to find solutions to those questions?

The (french) case against the NIH syndrome

Attention: les cascades de cette présentation ont été réalisées par des professionnels, n'essayez en aucun cas de les reproduire chez vous...

C'est en substance ce que j'aurai eu envie de lire comme commentaire de fin autour de cette très agréable soirée passée chez nos amis d'ABC Arbitrage, pour un meetup d'Alt.NET de haute volée, intitulé : "SOA et Service Bus" ce jeudi 3 octobre.

Pourquoi cela? parce que j'ai vraiment eu le sentiment ce soir là, d'avoir été complètement à contre-courant de l'enthousiasme général autour des messages délivrés par les speakers (mais il faut dire que l'excellente qualité de leurs présentations ainsi que leur état d'esprit très sympathique ont du aider). 

Mais avant de détailler ici les raisons de mon scepticisme, parlons d'abord des choses qui m'ont plues :

• Leur SOA pragmatique. En gros, l'important avec le SOA n'est pas d'utiliser tel ou tel protocole, ou tel ou tel format de message, l'important est déjà de penser son SI en terme de briques cohésives rendant des services avec un couplage lâche (le SOA apportant les mêmes bienfaits au niveau de l'architecture, que l'encapsulation au niveau de l'OOP)
• Leur découpage progressif et pragmatique de leur ancienne plate-forme monolithique en un modèle distribué et orienté service. On commence avec un service, on acquiert du feedback, de l'expérience ... et on continue petit à petit.
• Leur attention portée sur la réduction de la dette technique au fil de l'eau et des projets
• La cohérence de leur écosystème et de leurs APIs qui visent semble t-il la simplicité (l'approche "pit of success" qui est aussi une de mes marottes ;-) Le fait que cet écosystème cohérent facilite l'intégration et la productivité des nouveaux arrivants (à qui ils confient généralement l'implémentation d'outils pour compléter l'offre, comme première tâche)
• L'utilisation de produits open source, et leur "polyglot persistance", c.ad. de ne pas se limiter à une seule stratégie de persistance (i.e. SQL Server) en fonction de leurs besoins.
• La qualité de leurs supports et de leur présentations ce soir là. Très agréable et très efficace pour l'auditoire.
• L'état d'esprit positif, ouvert, dynamique et très sympathique de l'ensemble du crew ABC

Ca fait déjà pas mal de bonnes choses me direz-vous ;-)

Bon. Abordons maintenant ce que j'ai moins aimé:

• L'omniprésence des challenges techniques, et une absence quasi totale du métier dans leur discours. Je n'ai pas le souvenir d'avoir entendu parler de leurs utilisateurs, de l'interactions avec ceux-ci, des challenges qu'ils avaient à relever, etc. Du coup, cela a renforcé l'impression d'une équipe de DEV qui se fait un peu plaisir toute seule (et cohérente avec le NIH que j'aborderai plus tard). Je ne dis pas que c'est leur cas -pas de procès d'intention ici donc- je dis juste que cela a renforcé cette impression chez moi pendant leurs présentations. Car même si je kiffe la technique et apprécie particulièrement le mechanical sympathy, j'ai beaucoup de mal en revanche, lorsque celle-ci n'est pas pilotée par de vrais cas d'usages
• Le flou quant à la définition de ce qui peux justifier la création d'un service chez eux (quels sont les critères de création/d'ajout). Ce point, combiné à l'atrophie de la place du métier dans leur discours m'a presque donné l'impression qu'ils parlaient tout le temps de "services windows" et non pas de "services à portée fonctionnelle" (alors que ce devait être le cas à mon avis). Je pense que d'aborder un ou deux exemples concrets de ce qui était un service chez eux, aurait pu donner plus d'épaisseur et moins d'ambiguité à leur discours (en tout cas pour moi).
• La naïveté de certains choix : comme Cassandra pour une persistance "reliable". Car ce n'est pas parce que Cassandra est constitué en Cluster, qu'il est pour autant fiable et consistant (sur ce sujet, je vous invite à lire le récent post d'Aphyr qui explique tout ça, et à patcher vos versions de Cassandra au plus vite d'ailleurs, par la même occasion ;-)

Abordons maintenant ce que je n'ai pas du tout (mais alors pas du tout ;-) aimé:

• Pire que la justification, le discours d'évangélisation autour du Not Invented Here (NIH) syndrome (affublé ici de l'adjectif 'pragmatic') . Pour rappel, il s'agit de la tendance d'une entreprise ou d'une équipe, qui re-développe quelque chose qui existait déjà sous prétexte que cela n'a pas été conçu ou mise au point à l'intérieur de celle-ci. Que le contexte particulier d'ABC arbitrage (taille, budget, qualité de l'IT) les ai mené à faire tel ou tel choix d'architecture ou d'implémentation cela peut se comprendre (se discuter aussi, étant donné le côté quasi systématique de la démarche chez eux: hosting, déploiement, monitoring, middleware, ...). Mais promouvoir cette approche avec une certaine naïveté; j'ai été très, très mal à l'aise avec ce discours. Pourquoi est ce que je parle de naïveté ici ?!? Et bien c'est lié aux 2 autres points qui suivent:
• Le fait de minorer le coût total d'une telle entreprise (i.e. de faire croire à l'assistance que c'est simple, pas cher et rapide d'implémenter un middleware). En effet, implémenter une solution de messaging ne s'improvise pas, et ce n'est certainement pas en une seule fois que celle-ci sera mature et utilisable (et puis c'est du temps en moins passé à répondre aux besoins du business). Lorsque j'ai posé la question en séance sur le coût total de cette solution, c'est le coût initial qui m'a été donné (2 mois pour 4 développeurs si j'ai bien retenu). Et bien je gage moi, que le coût sérieux et total de la v1 (sur toutes ces années, et au fil de tout ces projets) n'a pas vraiment été pris en compte, lorsqu'il a été décidé de partir sur une implémentation de leur v2 (vs. acheter une solution du marché)
• L'absence totale de capacity management dans une telle entreprise (on parle du développement d'un middleware maison, tout de même!). Pour moi, c'était le plus gênant. En effet, comment savoir que l'objectif (de latences inférieures ou égales à la milliseconde, avec de la distribution de market data sur des 50aines de clients) est atteint  si on ne met pas en place des instruments de mesure concrètes ? Comment s'assurer qu'un affolement de la volumétrie des market data ne saturera pas totalement le réseau (ou la latence des publishers avant;-), étant donné la stratégie tcp-unicast choisie (vs. multicast), si on ne mesure pas ce qui se passe sur le réseau? En gros: comment savoir que toute l'activité de la boîte n'est pas mise en danger par certains choix d'architecture? Et bien, si on ne mesure pas : on ne peut pas savoir ! (sauf en le découvrant à nos dépens en prod --> j'aurais des dizaines de war stories à vous raconter sur ce sujet là...). Sur cette culture de la mesure qui me parait désormais indispensable d'avoir et de mettre en oeuvre, je conseille vivement l'excellente et très percutante présentation de Coda Hale: "Metric, Metrics everywhere")

En conclusion

Vous l'aurez compris, je suis beaucoup plus réservé sur le bilan très positif dressé par presque tout le monde du contenu de cette soirée. En état (et sans plus d'informations) je peux comprendre l'intérêt pour une société comme ABC arbitrage d'avoir eu recours à une telle stratégie de build systématique (étant donné la taille de la structure, et l'excellente qualité de ses développeurs), mais je trouverai très dangereux de généraliser cette approche à d'autres contextes (ce que je pouvais entendre en discutant le soir même avec les uns et les autres, ou en lisant certains retours très enthousiastes par la suite). Le plus gros risque avec cette approche NIH étant de perdre son énergie dans des combats annexes, et de ne pas se concentrer sur ce qui apporte véritablement de la valeur à nos utilisateurs (un travers contre lequel nous devons tous lutter, nous développeurs).

Pour ceux qui douteraient encore de la complexité d'une telle entreprise (i.e. développer son propre middleware low latency), je recommande la lecture de quelques documents produits par le Jimi Hendrix des middlewares, à savoir Todd.L. Montgomery (ça pour l'architecture, mais surtout ça pour l'implémentation).

En tout cas, et comme à son habitude, ce fut une excellente soirée ALT.NET, propice à de nombreux échanges et rencontres entre passionnés. 

Merci à Rui et aux gens d'ABC Arbitrage pour tout ça !

wiked! the solution to DRY all our projects KM

Since my latest post: It's really time to DRY our apps' Knowledge Management! I had the occasion to share this topic with several mates, and thus to discover that the maven site plugin was perfectly fine for what I intended to do (thanks to Christophe LALLEMENT & Alexandre NAVARRO by the way).

Thus, no need to reinvent the wheel for that (which is perfect fine for me).

So I started to play with maven (quite new for the .NET expert I am), and especially with the maven site plugin ecosystem. At the end, it took me several hours in order to fully understand how to use it properly. With some questions like:

• Which maven plugins (and versions) should I configure within the pom.xml I intend to use in order to generate the web site for my project (which is not a java project by the way)?
• Which minimal pom.xml could I set in order to generate a web site with the mvn site command-line (important for non-java project, and to KISS)
• Where should I put images and other resources in order to be able to reference them properly from my markdown pages
• Why and when should I have to call `mvn clean` before the `mvn site` command-line in some rare cases (like when changing the project name whithin the pom.xml file)
• How to set a better look n feel to the overall generated web site?
• Which markdown syntax was working, and which one was not well supported by the tools (i.e. the atx-style headers)?
• etc.

I finally intended to package something that would help me (and others) to quickly setup such web site generation solution for every new project Knowledge Management.

Thus, i'm proud to present you the wiked! solution,  now available on gitHub.

More than a simple bootstrapper that will ease you to earn some time when setup-ing such a solution for your project, wiked! also offers you a structure and some content for all your projects web sites (see the values and practices section for instance).

And since the DRY principle was one of my driver, I won't fully describe here what wiked! is all about. If you are interested, you can have a look at the wiked! ReadME.md

Hope this help.

The initiative that will probably shake the entire Capital Markets Industry...

A new Open Source foundation will probably change the face of the entire Capital Markets industry IT working model (sourcing, costs, time to market...). Its name? The Lodestone foundation.

Extract: 

The foundation will focus on delivering shared implementations of core services that Banks find themselves implementing time and time again. For example the foundation could deliver enterprise grade reference data platforms, market data platforms, OMSs, algorithm containers, trade repositories, grids for risk, pricing, etc. Some of those services could even be offered as a shared infrastructure, on the cloud.

The idea: to reverse the trend of innovation within the banking industry, by attracting some of the top  worldwide class experts you can find in technology today in an open source foundation dedicated to work on common financial blocks, and sponsored by some of the major actors of the Capital Market industry.

I bet that once the first tier-one sponsors names will be published, it will motivate lots of other financial actors (in order to have the opportunity to influence and participate with the foundation workgroups).

Check yourself the concepts with the nice sliderocket introduction of the Lodestone foundation during the latest Euromoney conference in London, and the Lodestone foundation web site (beta version).

Thanks to my friend Olivier, for having shared with me this amazing initiative.

Continuous delivery at Facebook

Chuck Rossi is explaining in a video how the Release management is handled at Facebook; it allows them to push daily updates to their site without production outage or service interruption.  This video is very informative and the announced KPIs are very impressive...

watch the video here

While we are talking about release management, I also highly recommand you to dig into the "continuous delivery" book and blogs of Jez Humble and Dave Farley.

Release management is definitively not only about process and tools; it is about changing our culture.

'REWORK': a must read by the founders of 37signals

'Meetings are toxic', 'Don't be a hero', 'Fire the workaholics', 'Interruption is the enemy of productivity', 'Build an audience', 'Good enough is fine', 'Go to sleep', 'Pick a fight', 'Planning is guessing' ... are some of the topics explained within the 37signals founders work manifesto.

This irreverent, but very concrete and clever book is definitely a must read.
It tastes like 'The Pragmatic Programmer' manifesto, but applied to the company level.

• http://www.amazon.com/Rework-Jason-Fried/dp/0307463745
• http://37signals.com/rework/