The Kitchen in the Zoo - management tag

How Brontosaurs kill Raptors

Wed, 01 Sep 2010 19:59:00 GMT

The board of a large organisation appoints a CIO who delegates technical decisions to his employees. In the Enterprise Architecture (EA) department, an expert on Business Intelligence (BI) decides that the strategy for this company is to have a single Enterprise Data Warehouse (EDW). This makes sense, because experience has shown that in the long run, it is cheaper and more efficient to build one Data Warehouse, rather than start by building "islands" and then later try to merge them together.

In the mean time, some business people working for the company have rallied the budget holders, and started a project to build a new online sales application. They need to generate some reports based on the sales from this application, in order to set themselves targets and measure their performance. But, they are not experts when it comes to reporting. The business sector at which their application is aimed is cutting edge and brand new, so they don't know what to expect, they don't really know what kind of reports they will need, and they will certainly need to adapt to the market very quickly.

A project team in the IT department then get the assignment to build the application, including the report generation. Also knowing little about reporting, they start talking to EA, who set up a meeting, between themselves, the development team, some people from the Quality Assurance (QA) department and some experts from the BI team. Hours of discussion later, they have made the following conclusions:

Company IT Strategy states all reporting must be implemented in the BI solution, to ensure there is only one EDW,
The BI team are the only experts of the BI system,
The BI team cost a lot, because they are using SAP and consultants (how many SAP guys aren't freelance?),
The BI team dictate a six monthly release cycle, no exceptions,
The BI system is a black box, its inner workings stay secret within the BI team.

So, a simple question arises... "Our customer doesn't know exactly what they want, and because of their business requirements, they need us to be agile. What do we do and can the BI team help?", the development team ask. The first answer skirts around the issue and doesn't actually deserve being called an answer, because it doesn't answer the question. The development team make the point that the question hasn't been addressed, to which another attempt to avoid the question is made. Clearly, either these people don't understand their businesses needs, or they know they can't help but won't admit it. Either way, the debate moves on.

Days later, the business guys are telling some colleagues about their cool new project over an espresso and a latte. "Hee hee hee, hoo hoo hoo...", laugh their colleagues. "We thought our reports would let us track our progress too, but the IT department screwed us over. They charged us double and delivered nothing useful. We tried to submit some change requests to get it working and all the IT department wanted was more money and they couldn't deliver the changes quick enough anyway!"

The cool business dudes ran over to the IT office, informed their development team about the new information and said, "Guys, we don't want you to use BI. Well not for the first release anyway. Maybe in a year or two. Sure, it's policy and all, but we don't know what we want, and need to gain some experience, so just hack those reports out within budget please!".

So the development team arrange another meeting with the ~~astronauts~~ EAs and QAs to get that unanswered question answered, and they talk about the customers concerns. Even if it's an over reaction, the point is, that another department needs to be involved meaning more communication, less visibility and hence greater costs and risks. "Sure, no problem if the requirements are stable and known, but this project needs agility", is the point the dev team makes. "Well, no, not a chance!", comes the response.

"In fact," they continue, "you simply need to go back to the business and tell them that their idea is an architecture infringement - it goes against company policies. You won't pass QA, you won't get the go ahead to go into production, and that's the end of it", comes the reply.

A solution oriented response indeed... "Furthermore," continues an EA, " the business doesn't have the right to come into our shop and tell us where to implement things and where not to. We are the experts in the technology domain. We make these choices."

Well, his argument is that the business hired the CIO, who delegated responsibility to a BI expert, who created a non-agile policy. So the business have to swallow the argument and accept that the IT department is shit and won't give them solutions which they need, to compete in new markets... I'm not sure he realises that this is his argument, but it is, if you look at it pragmatically.

I'm not sure the cool business dudes see it that way either. I know I don't see it that way. And I know of no solution oriented architect or developer who sees it that way.

Of course I understand the reasons we have EA, QA, Processes and Policies. For large organisations they are the only way to avoid utter chaos, amongst many other reasons. Most of my working life has been spent with large organisations, even though in many cases I was allowed to implement agile solutions. But, after a long day in the office, with some frustrating decisions, I need to rant, so here is my cynical view of EA, QA, Processes and Policies, albeit somewhat harsh: these are all things which get bigger, the larger a company is, and all things which let people hide behind their responsibility of providing the business with the solutions they need. They all cause large corporations to be compared to large dinosaurs. These departments seem to have forgotten one golden rule: if the business fails, those departments won't exist. In competitive markets like we have today, businesses, no matter how large, need to stay competitive, and they can do so by using IT departments which are able to deliver agile solutions. These departments also often remind me of self-perpetuating witch doctors (The Witch Doctors, ISBN 074932645X). They justify themselves by stating that the business learned the hard way and now that they have been built up they cannot possibly be removed or have power taken away, like we some how never managed to go live before they existed, or that small companies can't exist because they don't have them.

Copyright © Ant Kutschera

Social Bookmarks :

Houses of Cards

Sat, 10 Oct 2009 12:12:00 GMT

I have often heard of software systems being compared to a house of cards, meaning that they were poorly built and are ready to topple at any time. The system I am currently helping to maintain has from time to time also been labeled in that way, yet it always manages to go live with quite a few new functions, and apart from the odd all-nighter to fix a few last bugs, it works to the customers satisfaction.

So while I was doing the washing up today, I noticed something. I am well known amongst family and ex-house mates for building what look like unstable piles of washed up dishes, as they dry. Yet I have never ever had a pile collapse on me. Never once have I lost it all. And remember, there are slippery suds involved in holding up these piles!

As I thought a little more about it, it didn't take long to come to the simple conclusion that a pile of washing up is stronger than a house of cards. Cards are uniformly shaped and have no edges or surfaces which help to lock their neighbours in place. Plates, dishes, cups, pans and cutlery on the other hand can be placed tactically so that they lock together forming a strong structure. Notice in the picture below that the heavy pan is placed at the top!

It still looks ugly though. But as long as its just left to dry it will be safe. Just like our software that is left out in production where it runs and runs and delivers all the time.

Where my piles of washing up do go wrong, are when my inexperienced wife comes along and tries to take it down to put the dishes away. It has been known to collapse at such times (although luckily never spectacularly badly). Similarly, software needs careful hands and experience of how it was built, when taking it down and restructuring it.

Perhaps also relevant to software, is why I build such dodgy looking stacks of dishes. Well I find it the most efficient way. I simply start piling up in the order that the dishes come out of the bowl (please note that my software is designed and planned properly, unlike my dish washing!). What ever comes out of the bowl at that time, finds a good place to be anchored in. Pans get done last, since they are dirtiest and hardest to clean. Just like in software the hardest parts get fixed last because they take the longest to get right. They are put to the side with some water in them to soak away at the hardest parts. And when you are working to a deadline, you don't always have the time to rearrange your pile of dishes and fit those hard parts in, where they would look safe. You just build them in where they fit best. It looks unstable, but you built it and you know the history of why you did it that way. As long as you are involved in the next phase of the project or maintenance, everything will be fine (if you are not, it doesn't matter - see here). It's not ideal, but the software was delivered on time, on budget and it works. And just like with the washing up you don't have to spend your Saturday afternoon in the kitchen.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :

Creativity - Yet another driver towards agile processes

Mon, 28 Sep 2009 19:51:00 GMT

Software developers are often creative people, having entered into software development precisely because of its need for creativity during implementation. While solving a logical problem might sound unrelated to creativity, the ability to solve that problem in many different ways and choosing the right way, is what requires the creativity. This is what turns a mundane data entry job into an exciting and fun job.

Not only are there many different ways to implement algorithms or user interfaces, so that they provide the same inputs and outputs as required, there are many different ways depending upon the architectural or design view points taken. For example, should the implementation use as much reusable code as possible, or should the code be as simple, readable and hence maintainable as possible? Or is breaking architectural policy allowed, in order to ensure maintainability, productivity or other aspects, etc...

So if programmers, designers and architects are so creative, is it a problem? Well it can be, when requirements specifications are rigid. Developers need a certain amount of leeway in order to feel that they are being creative whilst doing their job. If a specification goes into exact detail about how to implement something (which a requirements specification shouldn't actually need to do, unless the requirement is defining the inputs or outputs of the system), then a developer can feel there is no need for creativity, and the fun part of their job disappears quickly. Once that happens, developer motiviation becomes reduced and project success starts to suffer. Indeed a measure of software success is the level of developer happiness at the end, because you need these people to be motivated for the next phase or project.

The alternative to rigid requirements is a more agile approach. If the customer* can work with the development team in an agile manner, defining the outcome of the software together and allowing the development team to provide input on say usability aspects, or technological options, then everyone is a winner. The developers feel creative and the customer gets something which he can definitely use and definitely wants, because the agile process is an iterative one with short cycles between feedback and response (delivery of an updated version of the system). Often when writing rigid specifications, they suffer from "specification rot" (exactly the same thing as code rot, which is when code quality reduces over successive iterations of changes because new changes fail to take old ones into consideration). Basically said, the specifications become incomplete, containing conflicting requirements and illogical portions. Close and agile work between customer and development team allows rigid specifications to become living documents which can be adapted and perfected.

An advantage to the customer is also that they may become aware of other possible solutions to their problem. For example there may be a cooler way to make the user interface interact with the users. "Cool" can be a very important metric when it comes to web site design. Don't believe me? Take a look at iPhone sales related to the "cool" factor. Developers may also offer better or more efficient ways to solving the customers problems.

Of course, there are two sides to any coin, so agile processes must have disadvantages. Here are a few:

    - Scope Creep - the customer realises quickly with each delivery that they need something other than what they have specified. But I have experienced plenty of projects where the customer doesn't realise that, until six months or a year down the line and by then all of their budget has been spent and it's too late to fix the problem. At least in an agile process, the customer specifies through priorities, what is most important to them within their fixed budget, and that priority list is allowed to change as the customer discovers the system.

    - Rigid release cycles - large corporations whose software is all interlinked may use say two fixed releases a year in order to synchronise releases and versions of software in order to reduce the complexity of running multiple versions of the same software at the same time. While such strategies do indeed make sense, there is no reason why individual projects may not be run lean and agile. The customer can certainly be given access to their software frequently as it is being built. The excuse that the software is not ready because interfaces to other systems in the big release cycle are not ready does not fly, because otherwise the software could not be built until those systems are completed (using stubs, mocks or similar are the solution here).

    - Reluctance to change - many customers who have aquired software in the past only know how to do it using rigid and old fashion methods. It is quite human to dis-like change and go with what you know. But I encourage you to try going agile on your next project, even if it is only to prove me wrong, that agile projects are more successful than rigid ones.

    - Inexperienced customers - to be a useful and good customer on an agile project requires that you are open to change and suggestion. Many customers create a picture in their mind of what they think they want and define that in a rigid way. They are afraid that if their rigid requirements are not implemented that the software will be wrong in the end. Well here's the news: it will be anyway! There are very few individuals out there who a) know what they really need without going through an iterative cycle of trying it and then improving it, and b) who even if they do indeed know what they really need, can communicate it so that it gets implemented that way. So instead of going through the disappointment of only playing with it when its finished, try playing with it as it comes to life, just like a new home owner inspects their house as it is being built, or an engineer tests their mechanics as they are built. In aeronautical engineering, the "customer" is generally considered to be the company designing the new plane, rather than airlines who purchase the planes. No aeronautical engineering outfit has ever built the entire plane before testing that the waste disposal systems worked as required! Indeed such systems are put through acceptance tests before they are even integrated into the airplane.

Out of these, the two important ones are inexperienced customers and scope creep / change. Prioritising EVERY TASK on the project is the key to keeping the project on track. Experienced customers know that arguing over the layout of the UI down to the last pixel while important, may be costly. They can decide if it is more important to the project (note: not them as individuals) where the priorities lie. Experienced agile developers know this too and together the customer and developers can ensure a successful project.

So, as the world moves on, there are still many non-agile and rigid projects happening throughout the world. If you are in charge of software requisition and have some budget to start a new project, why not try making the project an agile one, if only because the creativity involved makes it a more fun project!

*
Let's quickly define a "customer". In the sense of software development and aquisition, this group of people are the stake holders who are paying for the work to be done. They are not the users although should be part of that group in order to be able to specify what is really needed. In a world where a software product is to be built which will be sold to users, for example like what Microsoft's core business does, the customer here is not the end customer, but rather the group of people who decide what should be sold in the first place. The customer group typically includes business types who are in charge of the money and strategy, as well as people representing the user community.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :

NEWS FLASH: Agile does not mean there is no plan

Sun, 20 Sep 2009 19:53:00 GMT

Why aren't all projects run using agile methods? Working in an enterprise where we have maybe 50 applications which all go live at the same time twice a year (as well as another 450 which are also in some stage of the software lifecycle), I was told that it would not be possible to plan such a release without tight control over the planning, and frankly agile methods do not cater for such planning...

Erm... I see it a little different. Agile to me means that there is a plan and you know the maximum amount of money that you can to spend. To me it also means there is a fixed deadline. As quality is also fixed (come on, it has to be bug free!), it is scope which changes. Or admittedly to some degree also quality, as you decide that some bugs are tolerable and you can live with them.

What is different about agile projects, is that the project plan and scope (functionality) are alive. They are dynamic. They change from day to day as everyone on the project learns and discovers. Just like your day to day plans change as events unfold.

It should be illegal for a project manager to state that a project plan is fixed. It shouldn't be allowed that the project manager, upon discovering that a plan needs to change, states that it will not be changed because it has already been communicated to the customer. This view is certainly un-agile. But it is also simply wrong! It is like saying "I have stated my intentions to walk to the moon by Christmas, and I shall do so, regardless of whether it is possible". In my world, it is very acceptable to admit your plan is wrong, and to re-state your intentions, especially if the deadline doesn't actually change ("I shall now fly there instead!").

Transparency to the customer, that you have discovered something that you did not put in your original plan, shows the customer that he can have faith in you, for being able to be honest about it and to re-plan, being able to handle such events.

The customer must also be able to tolerate such unplanned events, and doing so shows that the customer is mature enough to understand that this happens in software development. It also happens through out life, in all kinds or projects. When was the last time that a building was built on time? Planning is done wrong in all walks of life, and it is perfectly natural to re-plan. We should not be scared of admitting that our plans are wrong and we should certainly not be working in environments where one feels the need to hide such facts.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :

The Key to Maintenance is Compartmentalisation

Sun, 21 Jun 2009 19:16:00 GMT

Over the last couple of large projects I have noticed a trend in the way software developers/engineers/architects like to come out with statements like "that code is shit". I'm not just talking about me saying it, or how they react to the code that I write (come on, I write great code!). But general sweeping statements like this crop up all the time on these projects, related to everyones code. And managers then perpetuate these statements.

I think the reason is easy to understand. When a developer takes over responsibility for some code they want to make it clear that any problems with that code are unrelated to them. They don't want to take responsibility for the code which someone else has written. Because writing code is inherently creative and the same functionality can be written many different ways, you are almost guaranteed that a developer taking over some code "would have written it differently".

What I have also noticed is that the developers coming out with statements like this do not know or do not want to understand the conditions under which the code was originally written. As an architect, I realise there are many many factors which influence the creation of code, budget being a major one. What some developers also do not fully appreciate is that the perfect implementation not only doesn't exist (because every developer would do it differently) but that it often isn't required. The perfect solution considers many cases which don't need to be programmed, as they "may" be required in the future, but currently are not. From a financial point of view it is possible to determine whether or not to implement functionality in software (see my previous posting). So from that basis, a developer should work to a simple 80/20 rule and implement something that will suffice, as opposed to something that will exist for the next quarter of a century, unchanged because of its greatness.

Often developers which make statements that the code they are inheriting is poor, want to refactor it. They give reasons that the enlightened manager/team-leader/developer understands that refactoring is part and parcel of modern coding. Or perhaps they give reasons like without refactoring or rewriting, the code is unmaintainable. But is that really true? Does experience show that once a project has gone into production that there are many bugs or many change requests? During maintenance what proportion of code will ever actually be changed because of bugs or change requests? The project I am working on now has some code it in which we have not touched in 18 months (since going into production). The code works, although today we do not fully understand that code, but importantly we have its source code! This code is without a doubt poor. It has relatively little documentation. We have a list of all changes that are likely to come in the next years (perhaps 5 years ahead), none of which a related to this code. The intended lifespan of the code is at least another 20 years. After 18 months in production there have been zero reported bugs related to this code. So should we refactor in order to be able to optimise the code and make it a nicer design so that we understand it today? I am of the opinion that we should not. Doing so would not add any value.

Indeed does a developer need to fully understand the code in order to be capable of maintaining it? I am of the opinion (and please feel free to disagree) that a good developer doing maintenance should be capable of maintaining code that they currently do not fully understand and which has been implemented using different styles/patterns/techniques than they would. In fact I would go as far as saying it is imperative for a maintenance developer to be able to do this. The reason is simple: to keep maintenance budget to a minimum. At the time of going to production, you almost certainly do not know the areas of code which will need maintaining due to bugs. You may be aware of future change requests or further project phases where changes will be made. The result is that a maintenance developer should not start refactoring code just to make it "more correct", unless that developer intends to support the software for the rest of its life. If that is not the case (and in my experience most developers move on to different projects within at least three years), then changing the code is useless as the next maintenance developer to come along will simply claim that the code is unmaintainable and poorly implemented and want to start the refactoring all over again.

The skill of being able to inherit code and appreciating that at that time you do not need to fully understand the code is in my opinion extremely important for reducing maintenance costs. The key is "compartmentalisation". A typical example in maintenance is that you will often come across code which is duplicated. The gut reaction is to refactor it because a "law" of OO is to have code reuse. How often though does it happen that during the refactoring you realise that getting the common code out is painful and not optimal? Often. You might even end up having to build in a work around (guaranteed to cause the next maintenance developer to claim you wrote crap code). And certainly, such refactoring may introduce new bugs, requiring it to be fully regression tested.

The solution then, is to "compartmentalise" the code. The maintenance developer should become at ease with the situation and make a mental note (or even in a erm... document) that this situation exists. Without knowing whether this code has a problem with bugs and without being able to anticipate any changes which may come, you cannot guarantee that the changes you make during refactoring will be optimal. However, by understanding such code and having several related "compartments" you can plan correct refactoring, should a bug or change come your way which affects that code.

What I am about to write now is controversial, but please read it with an open mind. If you make the assumption that your maintenance developers can compartmentalise and live with code duplication, then during the project phase where you implement your code, you can use compartmentalisation to your advantage: parallel development. If you have two developers implementing similar things and you get them to work together to create a single design, it will cost you more money. Firstly, because of the communication required (the positive communication as well as the negative whereby they disagree and spend time comprimising). Secondly because while one does the implementation of the common code, the other will be potentially idle. Thirdly, a common design may not be an optimum because of comprimise in order to make it fit both/all situations.

Hey, if worst comes to worst, at least if a bug turns up in on half of some duplicated code, the users will only see it once and not in every instance of some commonly used core code. There is a rumour in our office that aeronautical systems are developed parallel in duplicate by two teams who do not communicate. This ensures that critical systems can continue to run if a bug crops up in one of them... Of course you would actually need three such systems in order to be able to determine which one is actually displaying the bug.

Anyway, if you have got this far without claiming I am a nut case then congratulations for having an open mind. However please realise I am not advocating code duplication for the sake of it. As a designer or architect you should be in a position to determine where code duplication might occur and to plan it sufficiently correctly so that your team can work efficiently and use common components as required. But if your plan cannot efficiently include such a common component, or the complexity of it means that at the planning stage you are not able to determine the usefulness of common code, then don't get caught up in the "code reuse is imperative" mantra that OO brought us. When OO was invented, the idea of code reuse was first at the method level - OO empowered programmers to reuse functions very easily. Making a law that code duplication is illegal simpy constrains management/architects/designers from using their brain to consider the options (and be creative, which after all puts the fun back into software development).

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :