Why Software Engineering Isn’t Engineering, and the Implication for Deadlines

questionmarkSoftware engineering estimates and plans often fail to live up to the reality that follows. It seems to be the only engineering discipline in which this is the rule rather than the exception.

Before you leap to the defence of software projects you’ve worked on — perhaps even your own — think about how many times those projects delivered precisely what they said they would, and in the timeframe they originally promised. If yours did, you’re probably in a happy minority, or you’ve already embraced some of the realities I’m highlighting below.

“Engineering”… Really??!

Other than for extremely formal cleanroom and safety-critical systems, the flavour of software development undertaken in most companies isn’t really an engineering discipline at all, despite our ambitions to call it that. But the real problem is that we still predominantly estimate work and plan our activities as if it was, either within the software team or in our interactions with the wider company and its goals.

We architect and break down systems into subsystems, components and tasks, then estimate the resources required for each task. At the end of it all, we are usually surprised when our estimates don’t reflect the reality of what comes next. It is clear that the wider business is often similarly surprised and, therefore, no wonder that software estimates are often viewed with skepticism and disbelief.

Roads, Bridges, Bricks & Mortar

In physical engineering (such as civil or mechanical), estimates are based upon abstractions that are known to be quite accurate and dependable; component parts integrate and combine in ways that are by now well-understood. Each level is based on a foundation that is, at the lowest levels, ultimately based on well-researched principles: physics, chemistry, metals, concrete, glass, water, soil, gravity, etc.

Even the introduction of new modern materials is relatively risk-free as they can be tested, studied and understood in isolation or smaller builds, then integrated into larger builds with fairly predictable results.

That’s not to say that physical engineering doesn’t have it’s delays, complexities and flaws… but at least there’s the possibility of attempting to plan, reason and engineer them out.

Building on a Foundation of Complexity

Teams building software seem to ascribe the same physical characteristics to it as if it were more formal engineering. Whilst it is certainly true that there may still be physical laws governing the electronics and other hardware the software ultimately runs on, most software is built at a level of abstraction several hundred layers above that, and therefore on a foundation of potential complexity we may not understand quite so well.

Even where component parts of our system are well-understood (operating systems, languages, libraries), they can interact with one another, and with the foundations they in turn depend upon, in ways that components rarely would in a physical build.

Add to this the fact that most of our abstractions are about systems the precise likes of which we may have never personally built before, and it becomes clear that we are, at best, working with assumptions.

It seems we often mistake our ability to envisage a final system for the reality of the way that system will actually need to function, and the work involved in building it.

Complexity Appears Late

Problems and complexity appear notoriously late in software projects. Attempts to get an early understanding — such as proofs of concept, or use of spikes in the Agile world — can help, but they often don’t uncover the reality of the actual system we’re building… just one similar to (and therefore potentially totally unlike) it.

The result is that reality rarely adheres to our estimates and plans.

So What?

It is probably high time that we admitted most software engineering isn’t actually engineering, in the strictest sense of the term. The implication is that our estimates and plans — and our ability to hit deadlines — shouldn’t be relied upon in the same way. Embracing this truth, company-wide, might help us.

Secondly, I’d say that Agile, yet again, has something to contribute: We already know that it helps to nail requirements by working in an iterative and evolutionary manner alongside a customer, but it also assists in handling complexity and planning, by removing strict estimates and by working iteratively.

Most Agile “stories” are planned in points, t-shirt sizes, and other schemes, in an attempt to get away from time-based estimates and instead plan based on the perceived complexity and past ability of the team to deliver similar stories. Agile’s iterative approach means we can adjust course as complexity and other problems appear, just as we adjust course by regularly showing early versions of a system to a customer.

One implication of accepting this is that software projects will be notoriously hard to plan against calendar deadlines. This means the wider company will need to work with software teams in a way that depends less upon deadlines, estimates and plans. Many organisations embrace Agile in their software teams, but not in the wider company. This creates the untenable combination of Agile software work practices, but rigid / calendar deadlines and deliveries to the wider business… and the largely unacknowledged fact that these two worlds seldom align.

Maybe it’s time we embraced the reality of this less-than-formal “engineering” activity of ours, owned up similarly to the reality of our less-than-dependable “estimates”, and relied more on ways to work with these truths, such as Agile? This may finally mean embracing Agile in a truly company-wide manner, rather than just within software teams.

Hitting Software Deadlines: Essentials Only… then Upgrade

From “Deadline” can seem like a dirty word, in the software world. But without them, our work wouldn’t see the light of day, get an audience or have an impact. So despite their perceived unpleasantness, I’m all for them.

We tend to plan work towards a deadline in ways that ignore two inherent truths about software development, and therefore often miss them:

  • Complexity appears late: Despite our best efforts to surface it during planning and design activities. This is the reason plans go awry at the eleventh hour and deadlines that seemed achievable are inexplicably missed;
  • Completeness is hard to nail: Specifically, deadline-completeness. This means only the things that are strictly necessary for a viable release, launch or demo at the deadline. This is likely smaller than the list you feel you want to include, but adds crucial items that are usually not visible to a client, such as productionising, releasing, legal formalities, etc. Many of these are usually slotted in just-in-time before a deadline, or missed. Denial of the need to nail completeness, or discovery of further complexity here, can undermine a deadline.

Maybe we could try something along the following lines, to hit deadlines more often:

  • Find the Essentials for Deadline-Completeness: Begin by ruthlessly separating the work that must be completed for the deadline, but perhaps in a heavily-downgraded form.
    This might mean something releasable to a production environment (in a walled-off manner, minus a final trivial “reveal” on the deadline date), something that discharges legalities, and something that exercises the agreed use cases in a minimal but viable manner.
    Encourage people to write separate stories for bare bones deadline completeness, and then the rest.
    Also write meta stories for items required for completeness that would otherwise be forgotten due to not being visible to a client, such as deployments and productionising;
  • Hit deadline-completeness early: Complete the bare bones deadline-sensitive work as early as possible, and nothing else. In an Agile approach, this should be at least a sprint or two before the deadline itself. You will end up with a very minimal but released (though perhaps walled-off) system that would hit your deadline, but would perhaps disappoint you in a number of ways. Learn to accept this, because you won’t be revealing this version of your work and something much more crucial just happened that is worth celebrating: You already hit the deadline. You are already capable of delivering. So now, you can…
  • Upgrade, iteratively: Add in the work to upgrade that initial complete solution in tight iterations. Each of these should result in a system that is “better” in some way but still deadline-complete; remains releasable, still discharges legal obligations, still exercises the same use cases and perhaps some new and less crucial ones.
    Iterate on these upgrades, in a order of decreasing priority, until the deadline.

When the deadline comes, as it inevitably will, you are guaranteed to hit it in some viable and complete manner. There will always be upgrades you wish you could have included but they will be the least important ones and, much more crucially, you aren’t left with an incomplete solution that can’t released, demonstrated or used.

Approaching deadlines in this way means that complexity can be handled as it arises, without derailing the deadline. Hitting deadline-completeness in a minimal way, then upgrading, allows complexity to push out less-crucial upgrades if necessary. Yes, you don’t want it to… but the reality is it might need to. Post-completeness, there is room for the battle between complexity and upgrades to take place, however that plays out and without risking the deadline.

How do you persuade people to adopt this approach? By showing them that everything is still planned to be included, but that deadline completeness is prized first. Assure them that other upgrades, in decreasing priority order, will only be omitted if unforeseen complexities or problems arise. The maximum value possible is still going to be delivered. Nothing has changed but the approach to delivery and risk.

Isn’t this just Agile? If your approach to Agile (and there are many) includes a notion of deadline completeness, perhaps. But who really does Agile that way? After hitting the deadline early, the iterations are Agile-like. But this approach forces early inclusion of aspects that might be unseen to a customer in an Agile world — hence no stories — but which are crucial for the deadline, such as productionising.

Find the essentials and complete them early, then upgrade. By planning in a way that handles complexity like this, maybe we can let it do its worst — as it inevitably will — and still deliver to a deadline?

The Best-Laid Plans of Task-Based Engineering

018-linkedinWhen faced with the challenge of building software of any considerable size, we rather sensibly tend to begin by breaking down the complexity and fleshing out a design or architecture. Then we break down the work involved in building that design into tasks or user “stories”.

Sometimes we group stories into wider “epics”, themes or stages, but generally we estimate, schedule, and conduct the work of building software at a task level: Tasks are what we assign, place on agile boards, and discuss our progress in terms of.

(I’ll refer to everything as “tasks” from here on, but I also mean user stories, and much of what I’m referring to also applies to bug fixes and other ticketed items)

Tasks are abstractions of the underlying engineering work required, and a task-based view of engineering makes the process of building complex software more understandable, manageable and distributable as a team. But it also tends to ignore certain truths about software complexity, tasks themselves and the ways we actually work on them, both on our own and together in teams. In-particular, the tools we currently use to manage task-based engineering are rather restrictive and cumbersome.

So, given that it seems we need task-based engineering, what is it about the way we approach it, and current task-based tools, that can hamper our efforts?

Tasks Abstract Away Complexity

A task is simply an abstraction of the work involved in building a software feature, a use case, a component or maybe just of fixing a bug. As with all abstractions, it necessarily hides detail that, for all but the simplest of tasks, may be discovered later when we actually work on it, despite our best efforts at planning time.

Sometimes, as the true complexity of the work involved in a task becomes clear, we realise that we should restructure the task to break it into more manageable chunks, and that sub-tasks should ideally replace some (but perhaps not all) of the original parent task. This discovery of structure and complexity may occur several times, to an arbitrary depth, and perhaps only after part of the original task has already been completed. We may even discover aspects of the original design, and the tasks slated to build it, are wrong.

Rather than implying poor planning, this is simply a part of the discovery process and the complexity inherent in building software, particularly in an agile manner where we’re trying not to pre-think too much up-front. We need to accommodate that inevitable change, reflect it in the tasks we are working on and in those slated for the future.

Most current task management tools aren’t great at letting us change anything more than very small numbers of tasks and task-based planning. They certainly don’t help us to visualise the results we end up with. In a sense, they leave us with quite brittle and sizeable task-based plans. With some persistence though, it is usually possible to reflect the required changes, though this may require time that isn’t available and we risk the task-based view drifting away from the engineering reality.

What can also be particularly tough with current tools is arbitrarily defining sub-tasks to any depth, as we discover complexity and break down a piece of work further. We may need to have parent and child tasks representing slightly overlapping and part-completed functionality, whilst being able to see estimates for all of that work. I’m not sure that current tools allow much of this without significant discipline in our use of them. In this sense, they are too brittle to allow us to reflect the engineering reality beyond our (now outdated) initial plan.

Tasks Need Surrounding Context

Our planning process envisioned the whole (a delivery, a completed system, or a bunch of features for a sprint), then broke it down into tasks. Quite often, rather a bewilderingly large number of them. The tasks somehow lose our view of the whole, and it is hard to determine whether those tasks, collectively, still make up a meaningful delivery or omit anything crucial.

It seems like we need to retain some visualisation or record of the structure by which we arrived at the individual tasks, so we can work backwards from the tasks to the whole. Even when working on an individual task, we often need to get enough surrounding context to work on it, as each task rarely describes that in sufficient detail.

Being able to see where tasks fit into the bigger picture is particularly necessary when our plan inevitably changes over time, to avoid us getting lost in a changing sea of tasks, the intended outcome of which we can no-longer visualise clearly.

Current task management tools don’t allow us to retain the more complex structure of groups of tasks. We are limited to “epics”, tags, simple sub-tasks and basic categorisation. We need better ways of visualising how the tasks fit together to form the whole that we’re working towards, allowing us to spot omissions and to track progress. After all, the collective structure of the tasks will probably be as complex as the structure of the overall system we’re building.

Tasks Need Collaboration (Sometimes)

No-matter how much we break them down, often a task requires unexpected collaboration with someone else (together, or a hand off then a hand back), integration with their work or reaching an agreement. That micro-collaboration is hard to model, but crucial to a task’s completion, though we often assume (perhaps hope?) it is irrelevant.

Whether or not one person “owns” the task and the responsibility for its completion, several people may need to do work on it and must find time in their workload. This is particularly true with user stories, as we don’t tend to break them down into chunks any smaller than a story we can define in business / product terms.

Current task management tools suggest a single assignee per task, or multiple assignees but with no record of individual effort required or how the collaboration will occur. We actually need to reflect a more complex sense of ownership and collaboration, even if most of the work will be done by one person with fleeting collaboration at some point. Failing to capture this in a basic form can be a cause of tasks being unexpectedly “blocked” during sprints, or the logical next steps towards their completion being unclear.

Tasks Have Dependencies (Sometimes)

Sometimes, a task depends on other tasks being completed (or part-completed), or a bunch of tasks need to happen in a certain logical order. Often a task that is scheduled to be completed this week or this sprint is blocked by an implied dependency that won’t be resolved anytime soon. We must model dependencies, or risk impacting our progress if we discover them late, or if many tasks get blocked on a key resource that we should have planned around earlier.

Current task management tools can model dependencies clumsily at the task level, but aren’t good at helping us to visualise them. This can often be another cause of tasks being blocked during sprints. They certainly aren’t good at modelling dependencies between groups of tasks, and we tend to have to visualise those ourselves: e.g. All tasks in Stage 2 depend on the tasks in Stage 1.

A visual representation of dependencies at a task and group level, and ability to move them around, would give us a more efficient view on how best to approach those tasks in a sensible order.

Task Completion Often Isn’t Clean

Often a task is almost completed, with a small piece of work to be finished, perhaps when unblocked by another person or after a dependency or decision is resolved. The inability to easily mark the task as partially complete, or to define the remaining work (perhaps in a sub-task), hampers visibility of the true state of the task, particularly on agile boards.

Creating a new sub-task can be cumbersome, and it needs adding into the plan or sprint, in correct relation to the original task, so it isn’t forgotten or confusing when seen in isolation. Often the administrative effort leads to us just leaving the original task open until the remaining work is completed.

Current task management tools assume task completion is clean and atomic, leading to tasks remaining “in progress” when they are often 90% complete and the tool should allow us to easily reflect that. This often hampers visibility of remaining work, particularly close to the end of a sprint.

So What?

Clearly, I’m pointing out some broad problems with simplistic views of task-based engineering here and suggesting some very general solutions. Not all of these problems will apply to a project but, particularly with larger teams, I’ve seen many of them and their impact is real. Some teams tackle these problems well as part of the way they work, but others clearly struggle with the realities of task-based engineering.

A quick summary:

  • Tasks are valuable abstractions for understanding and managing engineering complexity, but the methods and tools we currently use to manage them are rather rigid. We need to be able to visualise and change large numbers of tasks easily, see their overall status and visualise how they combine to form a planned delivery or feature;
  • We need to be able to break tasks down to any level, describe what overlaps and what’s already complete, so we can model our fluid discovery of complexity but retain an ability to estimate and gauge the remaining effort involved. Again, current tools and methods are too brittle;
  • We need to be able to model and visualise task and task group dependencies, where they matter (they don’t always), or even just record that entire swathes of tasks must take place in a certain loose order. Current tools make this too cumbersome;
  • We need to model the necessary collaboration within individual tasks, both simple and more complex, so we can be aware of it. Current tools are too cumbersome to make this effective or even worth the effort, and agile boards are too simplistic.
  • Most crucially… All of this must remain light, easy to visualise and easy to change. Current approaches and tools are way too cumbersome, and often work against us.

As they say, “Your mileage may vary”. So I’d be really interested in hearing how this ties into your own views of task-based software engineering, whether you encounter any of these (or other) problems, and if you think the kind of broad changes I’m suggesting might improve things.

One thing is certain: Task-based engineering is here to stay!

The Mythical “Later” of Software Development

“We’ll add caching later”

“Let’s make it handle other data formats later”

“We don’t need to worry about that until later”

Let’s be honest… There is no “later” in software development.

In my 20 years coding professionally, I’ve never got to a point where I thought “Let’s go back now and do all of those things we said we’d do ‘later'”. That point in time never arrives, particularly on projects with deadlines, customers or real world objectives.

“Later” seems to be a story we tell ourselves time and time again when we’re busy coding, and we buy it every time.

But isn’t “later” just agile? – On agile projects and in lean startups, surely we aim to build only the absolute necessities right now, and everything else is “later”? Absolutely, yes! But I think we use “later” as a catch-all, a bucket, and a way of foreclosing on the need to (albeit briefly) decide whether we really mean something else.

Prioritise, backlog or possibly never – Whenever we say something will come “later”, we should try to be a little more specific. We usually mean one of the following:

  • Priority or Deadline – If “later” needs to happen by a certain release, deadline or just to be scheduled with a certain priority, then whatever we’re promising “later” should go into our planning system (JIRA, Asana, whatever we’re using). Let’s scribble it on a post-it so we can remain focussed whilst coding, then capture it immediately afterwards with a priority so it will come back to us automatically. There is no reliable prioritisation (or even tracking) system within our brain. So without capturing the task somehow, it will probably be forgotten and the deadline will pass, unless we add it to our…
  • Backlog – If there is no deadline for “later”, but it simply mustn’t be forgotten or needs more thought to decide if it’s even necessary, it should go into our backlog to be reviewed and prioritised another time. This is the list of items we must think about in our regular planning and prioritisation. Even if we eventually delete it from that backlog at some point, not adding it to the backlog now means it’s going to happen…
  • Possibly Never – The reality of most things we’ll do “later” is that they never happen. We never return to them or we rediscover them when it’s already too late. To be honest, this is ok for a great many of our “later” tasks. But perhaps we should just be honest about it. Sometimes, “later” is “possibly never”.

So I’m trying not to kid myself any longer with “later”, when I’m developing. It’s usually one of the above three instead, and it takes 2 seconds to figure out which.

And if I really mean “possibly never”, there’s a certain freedom in admitting that!