Being Agile… Part III: Failure is always an option!*

So we are TDD. We proudly announce the number of unit tests and the percentage coverage as part of the scrum achievements. We make demands on minimum coverage (for a brief while when we had TFS, it was a check-in constraint). But, what do we actually gain by testing? Is there a law of diminishing returns in testing?

Uncle Bob in his first presentation at our company demonstrated the bowling example. It is such a simple, eye opening experience to see how easy it is to over specify.

As I mentioned earlier, we were used to very delayed gratification. There were no demands on checking in code, actually we encouraged private branches for doing stuff. Sometimes it takes months before the changes could get in to the build.

The fun part about unit testing (especially if you have a “run this test” context menu) is the instant gratification, even in failure. You should first write a test that fails, so says TDD.

In a non TDD style of development we always expect to succeed. The first time I press build after a series of changes, I expect it to pass. The first time I run the application after a change, it better not crash. Even under the best circumstances a full build and run of the application, which was required if the change was in any of the core units, takes quite a while. Once the application comes up, we need to login and navigate to the view where the change is to see its effect. If we were to find something not working it would be a downer. So, we expected to succeed all the time, there by accumulating heart breaks upon heart breaks.

What changes when tests becomes the primary focus of development? When you write a unit test to model a new behavior, the first attempt is not even supposed to compile. Sometimes, if you are just fixing a behavior, we might have a unit test that can be compiled successfully without changes, but it definitely should not be passing. So, most times, we are specifically looking for a failure. It grows on you. I am no longer ashamed by a compilation error. It is a piece of information, sometimes quite valuable insight into the change that I am going to make. Since when a unit test breaks no airplanes fall from the sky (or angels die), we can afford to do this over and over. Every failing test gives us yet another insight into the problem, one more thing to do; every passing test makes us look for the next best way to fail.

Accepting failure as not just a normal outcome but as a desired outcome makes things much less stressful. If we fear failure, we will build safety measures for every imaginable way something can fail. The problem here is that, there are more imaginable ways to fail than plausible. And there are far more even plausible ways to fail than probable.

In a very fast pace environment things do go wrong from time to time. Since failure is welcome, there need to be a way to celebrate it. This is why we invented the Blame Gametm. When something goes wrong, when the build turns red, when a test “works in my machine” but no where else, when you wipe out the changes for 50 Fitnesse scripts because of one wrong merge, we blame. Of course, the blamee doesn’t have to accept it. There can always be come backs as long as they are more logically consistent and evidenced than “dog ate my hard drive”. The key is to embrace the failure.

What TDD, not just unit testing but aggressive acceptance testing teaches us is to fail often and fail gracefully. As we all know, if your millions of assertions never fail, they are as good as absent. The value of a test is when it fails.


*This is one of the best memes to come out of MythBusters promoted constantly by Adam Savage. There is one podcast where he describes why it is a fundamental principle for him. I hope it is not copyrighted by Adam or Discovery channel.

The Arsenic DNA Bacteria that might not be!

So, earlier last week, NASA comes out with an announcement saying that it will have a press conference on the 2nd of December to announce a major development that has implications to astrobiology.

A day after NASA announcement, Gizmodo published a highly speculative article that predicted that NASA is going to announce that they have found life in one of Saturn’s moons. Then there was just a flood gate of article, some of them with completely bizarre and unfounded speculations and rumors about the discovery.

On December 2, 2 PM, NASA announces the finding (at the same time the article was made available in Science Magazine titled A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus. The authors claimed in the paper and asserted in the press conference that they have shown that these bacteria can not only tolerate Arsenic but incorporate it into their DNA structure replacing Phosphorous.

This followed another flood of reports and blog posts. There was great excitement about the news, especially because it, if correct, would mean that the life can support a much wider range of conditions than we thought. It also will suggest that the main ingredients if life as we know are not that fixed after all.

First thing to come out of it was that, it was certainly over hyped by NASA by clearly hinting it is something more than what it is. Yes, finding a life form that can substitute Arsenate instead of Phosphate is an incredible finding. But, we know life on earth is very resilient and innovative. It would certainly don’t mean that we are closer to finding exobiological entities.

Along with this, there came a series of posts by scientists questioning the veracity of the methods used by the NASA scientists (Felisa Wolfe-Simon et al). Among the many I read about this subject there are two that stand out in its clarity of presentation and scientific rigor (no, that is not to make any claims about its correctness).

The article by Alex Bradley looks at the problem from a chemist’s perspective and brings up a few very serious process and interpretation issues. He correctly points out that there were due diligence that the NASA scientists did not perform.

An even more detailed criticism by Rosie Redfield can be read here. She goes more into the methodology and points out many potential pitfalls in them.

All this reminded me of the famous Carl Sagan adage “Extraordinary claims require extraordinary evidence”.

Another point this brings to light is the methodology of science and the immediacy and openness with which it is happening. Answering requests for comment, one of the authors of the article said that the discussion should happen in peer reviewed journals, not in the blogosphere. That is a bit odd though, considering that they themselves made an effort to make this very public.

Irrespective of how this particular finding turns out, science will go on. I have noted this earlier, the usually hidden away dynamics of the process of scientific enquiry is now spilling over for every one to see. It is beautiful, exciting. Now, if only the media learn to stop seeing everything in black and white.


ഇവിടുത്തെ ആദ്യത്തെ മലയാളം പോസ്റ്റിങ്ങ്‌!

കുറെ നാളായി ഞാന്‍ മലയാളത്തില്‍ ഒരു പോസ്റ്റ്‌ ചെയ്യണം എന്ന് വിചാരിച്ചുകൊണ്ടിരിക്കുന്നു. പക്ഷെ കൊള്ളാവുന്ന ഒരു IME എഡിറ്റര്‍ കിട്ടിയിട്ടാവട്ടെ എന്ന് കരുതി. അവസാനം ഇന്നലെ എന്‍റെ ചേച്ചി എനോട് ഒരു സഹായം ചോദിച്ചു, ഒന്ന് രണ്ടു ലേഖനങ്ങള്‍ PDF-ല്‍ നിന്ന് കണ്‍വേര്‍ട്ട്  ചെയ്യാന്‍ പറ്റുമോ എന്ന്. അപ്പോഴാണ്‌ മലയാള ലിപികളുടെ confusion ഒന്നുകൂടെ എനിക്ക് ഓര്‍മ്മ വന്നത്.

വര്‍ഷങ്ങള്‍ക്ക് മുന്‍പ് ഞങ്ങളുടെ ആദ്യകാല കമ്പനി ഒരു മംഗ്ലീഷ് എഡിറ്റര്‍ ഉണ്ടാക്കാന്‍ തീരുമാനിച്ചു. പക്ഷെ അതിനു വേണ്ട ഫോണ്ടുകളുടെ കാര്യം തപ്പാന്‍‍ തുടങ്ങിയപ്പോള്‍ ആണ് കാര്യങ്ങളുടെ കിടപ്പ്‌ ബോധ്യമായത്. നൂറു കണക്കിന് (ok, പത്തു കണക്കിന്) DTP സോഫ്റ്റ്‌വെയറുകള്‍ മാര്‍ക്കറ്റില്‍ അന്നേ തന്നെ ഉണ്ടായിരുന്നു. പക്ഷെ ഓരോന്നിനും അതിന്‍റെ മാത്രം ഫോണ്ടെ ഉപയോഗിക്കാന്‍ പറ്റുള്ളൂ. പലപ്പോഴും ഇത് സ്വാധീനം ഉറപ്പാക്കാനുള്ള ഒരു വഴിയായാണ് ഉപയോഗിച്ചിരുന്നത്. എന്‍റെ സോഫ്റ്റ്‌വെയര്‍ ഉപയോഗിച്ച് തുടങ്ങിക്കഴിഞ്ഞാല്‍ ‍ you are stuck with it.

ഏതായാലും ഒരു സംഗതി മനസ്സിലായി, ഇത്രയും കാലമായിട്ടും കാര്യങ്ങള്‍ക്ക് വലിയ മാറ്റമൊന്നും ഉണ്ടായിട്ടില്ല.

ഞാനും ശോഭയും കൂടി ഇന്നലെ രാത്രിയും ഇന്ന് ഇതുവരെയും കണ്‍വേര്‍ട്ട് ചെയ്യാനായുള്ള വിവിധ മാര്‍ഗങ്ങള്‍ തപ്പിക്കൊണ്ടിരിക്കുകയായിരുന്നു. പത്തു പതിനഞ്ചു ഫോണ്ട് ഡൌണ്‍ലോഡുകളും അസംഖ്യം ബ്ലോഗ്‌ പോസ്റ്റിങ്ങുകളും കഴിഞ്ഞ് ഒടുവില്‍ ശോഭ Google transliterate project കണ്ടെത്തി.

അവിടെ നിന്ന് IME എഡിറ്ററിലേക്ക് അധികം തപ്പല്‍‍ വേണ്ടി വന്നില്ല. എനിക്കറിയാം ഈ പ്രൊജക്റ്റ്‌ അത്ര പുതിയതൊന്നും അല്ല എന്ന്. എന്നാലും ചിലപ്പോള്‍ നമുക്ക് നല്ല ഒരു കാരണം വേണം കാര്യങ്ങള്‍ കാണാന്‍.

ഞങ്ങളുടെ തപ്പളിനിടയില്‍ പല മലയാളം മോഴിമാറ്റ സംവിധാനങ്ങളും കണ്ടിരുന്നു. പക്ഷെ അവയ്ക്കൊന്നും ഒരു IME എഡിറ്ററിന്‍റെ ഉപയോഗത കണ്ടില്ല. എന്തെങ്കിലും എഴുതുന്നതിനു മുമ്പ് ഒരു web browser തുറക്കണം എന്ന് വച്ചാല്‍ പ്രശ്നമാണ്. എനിക്ക് വേണ്ടിയിരുന്നത് OS-ഉം ആയിട്ടുള്ള സമ്പൂര്‍നമായ സഹപ്രവര്‍ത്തനം ആണ്.

കഴിഞ്ഞ അര മണിക്കൂര്‍ ടൈപ്പ് ചെയ്തതു നോക്കിയിട്ട് ഇത് പ്രായോഗികമാണ്, തികച്ചും. IME വാക്കുകള്‍ തിരഞ്ഞെടുക്കുന്നതിന് ചില കുണ്ട്രാണ്ടികള്‍ സംഭവിക്കുന്നുണ്ട്. കൂടുതലും ചില്ലക്ഷരങ്ങളും ചില കൂടക്ഷരങ്ങളും ഉപയോഗിക്കുന്നതില്‍. പക്ഷെ മിക്കവറും എല്ലാ സന്ദര്‍ഭങ്ങളിലും അധിക സമയം ചിലവഴിക്കാതെ തന്നെ എനിക്ക് ശരിയായ മാര്‍ഗ്ഗം കണ്ടുപിടിക്കാന്‍ കഴിഞ്ഞു.

മലയാള വ്യാകരണത്തിന്‍റെ കാര്യം വേറൊന്നാണ് – ഞാന്‍ മറന്നു പോയി. പല വാചകങ്ങളും എനിക്ക് മാറ്റിയെഴുതെണ്ടി വരുന്നുണ്ട്. അതുകൊണ്ട് തന്നെ ഇനിയുള്ള ദിവസങ്ങളില്‍ ഞാന്‍ കുറെ മലയാളം എഴുതാന്‍ പോവുകയാണ്.

For my non Malayalam readers, here is my language. There will be more like this coming in because I found this nice IME editor for my language (Google IME)

Being Agile… Part II: Never stop changing

The point I stop reading an article about agile development is when it starts quoting from the agile manifesto. No, I do not have any qualms with the manifesto; I think it is an excellent minimalist document. However, when people starts to preach about it, I tune out.

Same is the case when someone brings up a specific set of practices usually with a cute name as The Process. What I know from my last 7 years is, the only process that stays is changing processes.

This was very true in the beginning. We went back and forth and back to SCRUM as the organizing process, but played with its format and deliverables for quite a long time. We have tried weekly iterations as well as those that are longer than a month. We filled our walls with multi-colored post it notes. Built weird looking shared Excel sheets and used formal project management software with custom extensions to track the sprint.

We have been using unit testing to some extend even before our full plunge into the agile pond. But, the development and testing phases were mostly separated. After introducing SCRUM, and co-located teams, we were not sure how to interact with each other for a while. Lingering mistrust between the programmers and testers were quite palpable.

Most of our automated acceptance tests were UI tests. While these are in some sense the ultimate integration tests, precisely due to this overarching scope, they require most of a feature, including a near final UI ready before it can be run. In those early days of UI automation, the tests were extremely sensitive and just adding one control in a form would break a whole bunch of tests in mysterious ways. We actually had a campaign “does it break automation?”. (These days we ask, how can I come up with a breaking acceptance test, or a directed failure in existing tests to implement a feature or to fix a bug. More about that later.)

One of those days Michael Feathers came to our work and told us to go find fracture points and start clawing from there – I am paraphrasing. We have been looking at a really huge block of rock and wanted to see nice score marks, tap it with a soft mallet and there you get the nicely shaped pieces. He wanted us to look for fissures, cracks. That is what we did. Delphi, the language of our code base is not very refactor friendly. After 5 years of yearly releases, the original architecture was starting to form into a tangled web. (Hmm, so we have a huge rock with tangled web around it. See, I have my metaphor still going strong. How many of you have pulled the dry stems of climbers from rock surfaces? you have to be very careful or it will break. They do have a tendency to go into cracks!) Though we were a bit unconvinced about the feasibility of a test driven agile development strategy for our code base (we of course wanted to build from scratch!) we looked at our Java brethren with green eyes! They have all the tools, full reflection, managed code… We wanted it!

Once we started to pull at these stems, things started to happen. We tried to follow TDD as much as possible, but at the unit test level. However, once these frequent changes started to spill over and break automation, things became serious. Mind you, we are also working at break neck speed for a new release. While there was some consideration to the additional burden of process adoption, it did not change the deliverables much. So, if the automation is not passing, we cannot say a feature is done. If the feature is not done, we cannot get to stand in front of everyone and get an ego boost during the sprint retrospective.

Our early sprint retrospective started with a science fair-y demo of our features in a lab. We even got to sell our new ideas. After a few sprints though, we decided to do the demo in a formal fashion, power point or actual demo one at a time with a specified amount of time. Then we decided to do the demo at the team rooms and let the stakeholders walk from room to room. Then we decided to do it as a presentation for everyone, then we went back to team rooms, then we decided to record them and post them the previous day…

They all worked.

Coming back to the automation dilemma, soon it was clear that UI centric acceptance automation is not enough to support the new way of development. It was quite complex and time consuming to write and maintain. They also took awfully long to execute making it unviable to use as part of continuous integration. If we were to have some confidence in what we were doing – remember, much of the code we wanted to refactor were units that we seldom touched – we needed acceptance tests that are run with every build, or at least once or twice every day. Jealousy is a good motivation. We had been drooling over Fitnesse from the moment Uncle Bob showed us what it can do. There was no support for Delphi in Fit at that time, so we ported Fit to Delphi Win32 and started writing some tests. This was the same time when Delphi came out with a .Net version. We had to try it. We managed to compile enough of our code in .Net to allow us to cover the core and common business rules. This exercise to cross compile also created an opportunity to redefine layer boundaries by package restructuring. So we abandoned our win32 Fitnesse plan and started using the .Net version of the code to write Fitnesse tests for core functionality. This along with the Business Objects Framework that was introduced mostly through unit testing finally started to carve into the block making the cracks bigger and bigger.

We had a very supportive upper management during this transition stage. But, as the release progressed, each sprint, they will find some of the things that were supposed to be done was not done. This naturally brought up the question of accuracy of the estimates. Even though we were quite aware of the arbitrariness of the numbers we put in the remaining work column, it was never really sunk in. This gave rise to a series of estimation games and strategies. We had long planning sessions upfront. Longer planning sessions in the beginning of the sprint. More accountability of estimates. Complexity, risk and confidence factors, 0 to 1, 1 to 10, percentage… Attempts at accurate time reporting. And the all powerful velocity. We must have multiplied and divided every measurable and quantifiable aspect of the development process with every other to come up with a velocity unit. Ideal team, developer days, 2 hours allowance every day for meeting, fractional contributors.

They all worked…

Even when many of them did not bring forth the result we hoped for. But, when they didn’t, we got a chance to learn why it did not. Isn’t that the spirit of any scientific enquiry!

Being agile… Part I

The noise level about agile software development is deafeningly high these days. May be it has already peaked, which is probably a good thing.

My real encounter with agile development in a production environment happened in 2003, when the company I am working for decided to adopt agile practices. We were playing with Fish philosophy before that. That was quite amusing and often gave me of an Office Space feel. It was like someone was trying to pour happiness down my throat.

When I was sitting in the early presentations and crash courses on agile, I was quite skeptical in its adoptability for our code base. Our experience of the first three years of agile was presented in the 2005 agile conference (Teaching a Goliath to Fly). 5 years after that paper, we are still agile, more so.

So, among all the other noise, I will add mine as well.

I am planning to write a series of posts about the interesting facts, realizations and revelations during this time, my reflection on the larger state of affairs etc.

First of all, I believe the spirit of agile development is its lack of rigidity. Unlike the earlier, well defined software development life cycles, which specified (sometimes including visual/text format) of artifacts that are to be used in each stage of the development, agile presents some basic principles. There, of course, are attempts by many to present such over specified artifacts in agile as well, but, it is an exception, not the rule.

It all comes down to the realization that software development process is quite messy. But any complex human endeavor is messy, especially ones that involve a lot of abstractions. In many engineering practices, we have managed to come up with systems and practices that controls this messiness to a very large extend. However, if you have ever associated with a construction project, it is easy to realize that with all the plans and architectural drawings and RFPs etc., the final product turns out to be quite different from our original conception about time, resources and form. But, since we know the costs involved in making a change, we just try to live with it.

In case of software, there is a common assumption that it can be altered quite easily – that it is soft, and malleable. It is also quite abstract even in its final form. It is a model of the real world, a simulation of a series of behavior patterns of the store front clerk, physical movement of a bunch of trucks. They communicate tersely with the user, mostly in verbal, or in highly iconized visuals. We create this model through a series of layered abstractions from real world observations, verbal descriptions, mathematical equations and finally the implementation tools (programming language, testing tools, modeling tools etc.)

Many early attempts at controlling the messiness of software development did so by controlling change. Mimicking other engineering disciplines, we tried to create detailed design artifacts, elaborate triaging procedures for change control and sometimes downright scare tactics! Every stage of the development created these huge walls of artifacts between its predecessor and successor. And as with any wall, it seeded animosity. I remember, in my previous job, I met an actual tester only after several months. But, even before meeting one in flesh, I was quite happy to trash them and developed quite a distaste in their ways. We did the same to the “architecture team” whom I never met in my two years there.

A significant symbolic gesture we did, prompted by Ken Schwaber was to demolish the walls of our cubes. Our cubes had 5ft high walls and you could only see the person sitting right across you. The window cubicles were a status symbol. It also separated the programmers from the testers and from the BAs quite effectively by placing them in different areas of the floor. It was a shock to a lot of people to lose the privacy of their cubes. Many complained the new “team rooms” are too noisy, Ken complained that we are too quiet. (The team I am in now has the honor of being the noisiest!)

We were officially following SCRUM and agile. But, the nature of our products made this adoption quite challenging. Since we are developing packaged software, there is not a lot of direct and immediate interaction with the end users, and the release cycle is typically span an year or more. The adoption of new versions by customers is even slower. There were serious doubts about the predictability of iterative process. We were changing things so very often. Trying new ways of planning, inventing complex sticky notes schemes, pairing and not pairing, fighting over differences between unit testing and acceptance testing.

One thing we did not do was adhere to a single set practices.

Chevy Volt has WHA!?

It has 10 million lines of code, exalts Wired magazine, making it the King of software cars.

Now, if we look at kings in real life, who are overly pompous, over spending, ego maniacal AHs, this title may suite.

Oh, no, I have nothing against Volt. I would buy one (or another Plug-in) when it is time for my next car. But there had been stories similar to this taking rounds in the web. It looks like the manufactures want to make this a story. Look at us, we have 10 million lines of code!

First of my problems with this statement is, lines of what code? Since lot of the code are controller and other microprocessor code, it could be just instructions. In that case, 10 M instructions is not that big a code base. It might be a good thing to have only that much. But if this is 10 M lines of code in some high level language like C, things look much different. Then the question is why the heck so much lines of code!

There was a time when people used to boast about the numbers of lines of code in their code base. There were even places which used to pay per lines of code. But, if someone these days try to bring up the number of lines of code with a sense of achievement, unless it is to show how few lines there is, it is unimpressive.

A lot of people assume that all the work in creating a software product is in finishing it up for the first release. But, the fact is, it is only the beginning of work. Not just fixing bugs but also keeping up with user demands for new features, accommodating new scenarios etc. So, the total cost of a software development process is overwhelmingly decided by its maintenance costs.

That is why the best software product is the one that does not have any lines of code. So, Chevy Volt is 10 million times worse than the theoretical best Open-mouthed smile

Fallacy of Exaggeration

In an article about the brain (a wonderful article) Carl Zimmer has this quote.

The brain is, in the words of neuroscientist Floyd Bloom, “the most complex structure that exists in the universe.”

Now, I understand the awe we feel looking at the complexity and ingenuity of our brain, but the most complex structure in the universe?! Biological systems tend to accumulate entropy to create complex systems, but there are many other phenomenon in the universe that has much more entropy. Think about the nuclear fusion in the center of starts, and how about the super massive black hole! Now think about possibilities of life (as self replicating, self regulating systems capable of building complexity) in the rest of the universe. Some of it might have had billions of years to evolve. May be there are sentient networks that cover whole solar systems.

How do you even conceive even a thought experiment to verify that statement?

I have to disclose that I did not read Bloom directly and doesn’t know the larger context from where it is taken.

Such statements, like any other unverifiable statements, should be avoided when someone writes about science.

My Dinner

We both like cooking and eating good food. We both agree that we can cook better food than most restaurants. And our cuisine is pretty eclectic. However, some days we fall back to the nice Kerala style dinners. Today was such a day. When everything was on the plate, I couldn’t help but take a picture. So, here it is

My Dinner

Clockwise from left:

Green cabbage stirfry, green eggplant stirfry, snow peas stirfry, sardines shallow fried in their own oil, chick peas stir fry, tender mango pickle. In the middle of course is Kerala brown rice with sambar and curd (Yogurt).

What is not evidence!

In most discussions that I have with people of fervent religiosity, one thing that frustrates me is the lack of basic understanding of what constitutes as evidence. So, today I found this gem of a list in the comment section of a blog. It is a pretty comprehensive list of types arguments usually used from the religious people to defend their faith.

Allegations are not evidence.
Hearsay is not evidence.
Unsubstantiated claims are not evidence.
Personal revelation is not evidence.
Anecdotes are not evidence.
Rumors are not evidence.
Wild speculation is not evidence.
Wishful thinking is not evidence.
Illogical conclusions are not evidence.
Disproved statements are not evidence.
Logical fallacies are not evidence.
Poorly designed/executed experiments are not evidence.
Experiments with inconclusive results are not evidence.
Experiments that are not and cannot be duplicated by others are not evidence.
Dreams are not evidence.
Hallucinations/delusions are not evidence.
Experiments whose methodology is not open for scrutiny are not evidence.
Data that requires a certain belief is not evidence.
Information that is only knowable by a privileged few is not evidence.
Information that cannot be falsified is not evidence.
Information that cannot be verified is not evidence.
Information that is ambiguous is not evidence.

Now, only if everybody play nice and follow these basic principles. I am not very optimistic. Because, if the theists, especially the fanatical one used any kind of logic and rationality, they will find their own arguments crumble like dry Puttu

Create a website or blog at

Up ↑