“Don’t worry about the details”,… said no software developer ever

bubbles1bDevelopers spend our days deep in details, usually a thousand and one of them. Combining and orchestrating them up to the point where they collectively deliver something tangible to other — often non-technical — people.

Up and down the stack we go, from the bigger picture down to concrete implementation details, and back up again.

Good software developers can traverse that stack all day long; speaking to people or explaining concepts at all levels… in language appropriate to that level, using abstractions and terminology that hide the detail below, but always being aware that the detail is still there,… waiting to be dealt with.

We know the bottom line is that the details have to work, or there is no bigger picture.

The trick is learning the right times to get into those details, how deep to go, and when to acknowledge that problems at one level have an impact on a wider scope. What we try not to do is get lost in the details in ways that aren’t relevant to someone who just wants to know about the overall solution or a particular layer of it. We try not to baffle you or drag you down the rabbit hole with us,… and sometimes we succeed.

In most other professions, you operate at one level and you remain there. Not so with software, where we work with the details but orchestrate and explain them at a level appropriate to many audiences, whatever they might need. We deal with customers and other developers, and everyone in between.

So if you ask a software developer about a solution or talk to us about a problem… we’re usually traversing the stack as we talk, picking out what’s relevant… choosing language… hiding details, unless they impact you… storing up things irrelevant to you that need investigating later… trying to be open without being complex,… honest but not alarmist. And if we’re good, you can’t tell we’re doing it.

The details matter to all of us eventually. But it’s our job, as developers, to figure them out.

The Introvert and the Startup

feetPerhaps nowhere is the phrase “one size fits all”  more misplaced than when talking about work environments and company culture. This is particularly true if, like me, you just happen to be something of an introvert.

At startups with aggressive growth and targets to hit, we try to foster a culture in which people have the opportunity to contribute their strengths and abilities, whilst creating an environment in which they want to spend a significant amount of their time. We also want to have some fun to offset all the hard work.

The creative options and flexibility for doing this at small startups simply can’t be matched in the wider corporate world, so it is no wonder that startups tend to have some of the most enviable work environments and cultures.

But getting it right involves a complex interplay of factors when catering for a diverse workforce, and talented people bring their whole personality to work. Tweak one environmental or cultural aspect to suit certain people or to get results, and you often find you’ve ruined another aspect for someone else. Worst of all, you may even hamper their ability to contribute.

There is perhaps no group that this applies to more than introverts, as many startup environments can be somewhat charged up and geared towards the high-energy needs of extroverts.

introvert, noun /ˈɪntrəvəːt/

A person who is more energised and stimulated by spending time alone or in low-key situations, than with others, and who finds certain group interactions exhausting.

To my amusement, I find it easier to tell people that I’m gay than to admit being an introvert. The former rarely causes a ripple. But the latter is often the start of a grand inquisition: “What’s an introvert?” “Why do you say that?” “You don’t seem like one” “We don’t need introverts here” “You just need to come out of your shell and let your hair down”… All precisely the kinds of confrontation an introvert wants to avoid at any cost. It’s no wonder we usually try to fly under the radar.

It is said that people are hard-wired somewhere on the introversion / extroversion spectrum, or we learn it very early on in life and it can be incredibly hard to unlearn. This has certainly been my experience. But what we are rarely told are the upsides to being introverted, particularly in environments you might think it would be a weakness, such as at startups. Indeed, a mix of introverts and extroverts may be the ideal team configuration… but only if handled thoughtfully.

So what are some of the unexpected strengths of introverts?

  • Leading – Many famously great leaders report being introverts; Bill Gates and Abraham Lincoln among them. Perhaps introverts make good leaders because it didn’t come naturally to us, we had to learn it and tend to have a more considered leadership style that we adapt to the needs of those we are working with. We also have less of a tendency to micro manage, choosing instead to let people exercise their abilities and prove themselves to us.
  • Presenting, Explaining – Strangely, I have no difficulty standing up in front of 200 people to give a presentation or to explain a technology. But after presenting, being part of the audience as a crowd can suddenly seem daunting to me. This is perhaps the opposite of what you’d expect.
  • Listening, Insights – Introverts tend to develop good listening skills. We have a tendency to step back from crowds, or to remain at the edges. We also spend more time processing what has been said to us, less time interrupting and, because we naturally need breaks from engaging, we are more likely to spot connections and insights.

But what do introverts find harder, and what might we need adjusting in work environments and culture so we can thrive?

  • Discussion, not Confrontation – Discussions that turn into confrontations, or environments that verge on being boisterous, can be tough for introverts. We tend to retire rather than engage, feeling shut down rather than fired up. The precise opposite of how extroverts react. When discussions stick to facts, rationales and explanations rather than the raising of voices, introverts can actively and even energetically engage and, as mentioned above, may have an entirely different set of valuable insights to offer. Conversations can still be very animated and productive.
  • Space to Recharge – We tend to need time in which we are likely not to need to engage with others, so we can digest, consider and recharge. We also need the physical space to do that in. If an environment is socially charged non-stop, introverts can rapidly be worn down, drained and become detached.
    Good environments allow a mix of engagement and focus, with meetings and work spaces kept separate where possible. Some would say this limit on engagement is a downside, preferring constant communication instead, but I’d say it naturally timeboxes interaction and forces more considered work to be scheduled. No-one, not even extroverts, can stay engaged, communicative and effective the whole time. Open-plan spaces can be a drain on everyone, not just introverts.

So perhaps the strengths we’re recruiting for in fast-paced startup environments might occasionally come aligned with a personality trait we’ve usually considered to be a weakness: introversion.

Introverts comprise up to one-third of the total workforce, so getting things right for a mix of personality types could be a significant win for startups looking to build great teams and great places to work. And if startups can get this right, the wider corporate world might follow.

Headspace: Perhaps the Most Urgent Thing?

londonNothing of any significance or impact was ever created by a scattered mind.

Great things are not built in a rushed series of truncated thoughts, punctuated by distractions. They are created by getting the headspace to go deep, to stay there a while, and to surface with new insights, structure and conclusions.

Even with a eureka moment, a spark of brilliance or a chance connection, those initial insights take focus and refinement to develop, distill and simplify; to make them usable.

In fast-paced modern work environments, it’s clear that most of us aren’t managing to achieve headspace or to go deep into our work with any regularity. Our days seem like a series of distractions, email, phone calls, chat, meetings and being ever-available. Busy-ness, and particularly visible busy-ness, is a medal we seem to wear with pride.

But it’s likely that we’re barely scratching, then re-scratching, the surface of the work in front of us. We’re probably repeating large portions of it, as we absorb the cognitive switch away and back again. If we’re lucky, we’re gradually making progress in the right direction, though it will be hard-won, inefficient and lacking any real insight or depth. We may find ourselves achieving most of our meaningful work outside of office hours, which implies our work environment isn’t… well, isn’t working.

Most worryingly, we may be killing our ability to innovate, to compete and to deliver value… which is surely why we’re at work in the first place?

After 20 years working in teams, I spent a year working alone on two startup ideas… yet initially I still found it hard to focus. Surely, I had assumed, headspace would be simple to achieve alone?! It seemed that the learned expectation of interruption and the numbing array of possibilities to do other lesser things to get a hit of achievement kept me from focussing deeply for any duration. I had become a busy-ness and achievement junkie. At first, I seriously wondered whether I had some kind of attention deficit problem and so I turned to tools like the Pomodoro Technique to regain my headspace.

I gradually learned to commit to 25-minute periods of focused work, during which I would let nothing else (including my own thoughts and other interests) distract me, and then to take a short break or work on something less involved. Before long, I could crank out 6-10 of these “pomodoros” per day without too much struggle, and my ability to focus on my own returned.

During that year of regular focus, the work I produced felt significant: I made leaps forward, created two products and wrote my own UI framework. These achievements would never have been possible in my usually-scattered frame of mind. All of them were the result of focus and depth… of finding regular headspace.

Even more so in teams, the ability to carve out periods of headspace is key to getting work done together. But balancing it with being available on-demand is also crucial. We simply can’t schedule the needs of other people and the wider team, nor block them out to pre-determined times. But perhaps, after a landslide of unscheduled requests, we can push back a little, pop on a pair of headphones (a visible sign of focus), and get 30 minutes of focused work done before accepting more outside input. Oscillating in this way between being available and being focussed allows a portion of our time to be allocated to deeper work, even if we have to be a little creative and flexible about precisely which hours of the day that ends up being.

Respecting Headspace: It may seem that getting immediate answers from colleagues, and being available to provide them ourselves, keeps a work environment moving at maximum speed. But there is a cost to the team, and therefore to the project, for that immediate answer, paid in terms of the headspace of the person providing it. It may feel as if asking in-person is more productive and that chat or email are inefficient ways of getting an answer, but those tools allow the person providing the answer to crank out a series of such answers during a quick break from other work, and perhaps still within a relatively short timeframe. More urgent interruptions can still take place, but they must be balanced with the permission to block out further interruptions for a period of time that each person chooses themselves. Our headspace and their headspace will rarely occur in-sync.

Deserving Headspace: There is a flipside to this. In order to be allowed to focus, we must also prove ourselves capable of handling those queued requests in a responsive and reliable manner. Ignoring emails and not scanning chat channels for updates occasionally are certain ways to force people to interrupt us with more urgency, to grab our attention and to destroy our focus. If people learn that we will always respond within a reasonable timeframe, they will happily learn to queue non-urgent work for us.

Respecting the headspace of others may improve the output and depth of work produced by the whole team, by reducing cognitive switching. It probably also doesn’t greatly delay the answers our interruptions require, and it can be mixed with much more interactive sessions, standups, meetings and brainstorming to get the best of both worlds.

Headspace in teams, but also alone, isn’t easy to achieve. But what’s clear is that without it, the work we produce and the things we create will be mere shadows of what we’re capable of.

Maybe headspace and our ability to focus really are, somewhat counterintuitively, the most urgent things of all?

 

Image: (c) 2015, Ian Cackett

 

“Someone” Doesn’t Work Here Any More

someone

“Someone should fix the…”

“Someone will need to…”

I hear it all the time. I even say it. There is always so much for “someone” to do.

But “someone” is a problem: A cause of forgotten issues, broken links, missing pieces, out-of-date documents, poor integrations. “Someone” clearly isn’t working.

“Someone”, when uttered, tends to mean “not me”. A way of shrugging off the item, of distancing oneself from it.

We need to ban “someone”, and instead…

Find “someone”… or a team. Or nominate one. We should make sure they will take on the issue, or put it on their list, without dumping it on them. We should check they will own it. We should look on it like a baton handover in a relay race, rather than the throwing of a grenade from a-far.

Make “someone”… if there is no-one specific. We should make a placeholder list for issues and items in the same category. We should make sure a person or team picks them up at the right time, or keep a prominent record of those lists for review. We should make sure they are easily found and not lost. This is often necessary before hiring the first person in a department, such as marketing. A placeholder list might be needed for that team, until they exist.

Become “someone”… The ultimate, is to become the “someone”, for issues we could tackle or own, or just be willing to hold until a more appropriate “someone” can be found. Sometimes this means becoming “someone” in conjunction with another person, because these are often the difficult items that require collaboration and pushing forward to completion, particularly if they fall between the remit of several teams.

In rapidly-growing startups, particularly, we all collectively need to step into the gaps between teams, the places that aren’t owned by anyone or any process yet, and fill them. There can be a lot of “someones” in startups, if we aren’t careful.

The truth is, unless we are all “someone” from time to time, then the task will default to our other favourite friend, who is always more than willing to take it: “No-one”!

The Three Hardest Words: “I Made This”

dish“I made this”, is what we say by implication when we publish, send, launch, perform or unveil work that is entirely our own. Three wonderful, difficult and daunting words.

We are the only ones who can choose when — or whether — to say them, whether to hit “publish”… whether to take the risk.

Solo entrepreneurs, writers, bloggers, artists, performers, artisans all know the feeling. But even in a team at a small startup, we may feel this to some degree because we may singlehandedly represent some entire aspect of a company, a product or a service.

Our work represents us. Once we unveil it, it is out there and usually can’t be retracted. There is no-one else to stand behind it, to share the blame or the praise. No large corporation to take the flack. No anonymity. It’s like publishing a condensed version of ourselves; a snapshot of what we are currently capable of, warts and all, for the world to see.

It takes nerve to accept feedback – good, bad and indifferent – and still stand by our work, to discuss potential improvements to it, or the inevitable choices we could have made. There is no perfect launch and nothing is ever truly “ready”.

The crucial thing is realising we did our best at the time, that no-one is 100% correct, and still being glad that we put it out there. It takes guts to realise we will inevitably want to retract, delete, smash it to pieces, or to endlessly correct it based on feedback — or the perceived feedback of indifference — but to keep it out there anyway, in its original form.

So far, I’ve done this with two software products I developed alone, this blog, photographs and other articles. I have friends who have done it with startups, artwork, experimental new food products, books and performance art. I’m proud to count myself among them because I think people who have done this are amazing.

I’m convinced that we learn more by putting our work in front of an audience than we possibly ever could by thinking and agonising about it in private, by tweaking it and waiting until we feel like it is somehow worthy or “ready”. Those lessons feed back into our next attempts, but only if we make our first attempts.

If you’re in doubt, it’s probably ready. So take the leap, and say “I made this”.

Why Software Engineering Isn’t Engineering, and the Implication for Deadlines

questionmarkSoftware engineering estimates and plans often fail to live up to the reality that follows. It seems to be the only engineering discipline in which this is the rule rather than the exception.

Before you leap to the defence of software projects you’ve worked on — perhaps even your own — think about how many times those projects delivered precisely what they said they would, and in the timeframe they originally promised. If yours did, you’re probably in a happy minority, or you’ve already embraced some of the realities I’m highlighting below.

“Engineering”… Really??!

Other than for extremely formal cleanroom and safety-critical systems, the flavour of software development undertaken in most companies isn’t really an engineering discipline at all, despite our ambitions to call it that. But the real problem is that we still predominantly estimate work and plan our activities as if it was, either within the software team or in our interactions with the wider company and its goals.

We architect and break down systems into subsystems, components and tasks, then estimate the resources required for each task. At the end of it all, we are usually surprised when our estimates don’t reflect the reality of what comes next. It is clear that the wider business is often similarly surprised and, therefore, no wonder that software estimates are often viewed with skepticism and disbelief.

Roads, Bridges, Bricks & Mortar

In physical engineering (such as civil or mechanical), estimates are based upon abstractions that are known to be quite accurate and dependable; component parts integrate and combine in ways that are by now well-understood. Each level is based on a foundation that is, at the lowest levels, ultimately based on well-researched principles: physics, chemistry, metals, concrete, glass, water, soil, gravity, etc.

Even the introduction of new modern materials is relatively risk-free as they can be tested, studied and understood in isolation or smaller builds, then integrated into larger builds with fairly predictable results.

That’s not to say that physical engineering doesn’t have it’s delays, complexities and flaws… but at least there’s the possibility of attempting to plan, reason and engineer them out.

Building on a Foundation of Complexity

Teams building software seem to ascribe the same physical characteristics to it as if it were more formal engineering. Whilst it is certainly true that there may still be physical laws governing the electronics and other hardware the software ultimately runs on, most software is built at a level of abstraction several hundred layers above that, and therefore on a foundation of potential complexity we may not understand quite so well.

Even where component parts of our system are well-understood (operating systems, languages, libraries), they can interact with one another, and with the foundations they in turn depend upon, in ways that components rarely would in a physical build.

Add to this the fact that most of our abstractions are about systems the precise likes of which we may have never personally built before, and it becomes clear that we are, at best, working with assumptions.

It seems we often mistake our ability to envisage a final system for the reality of the way that system will actually need to function, and the work involved in building it.

Complexity Appears Late

Problems and complexity appear notoriously late in software projects. Attempts to get an early understanding — such as proofs of concept, or use of spikes in the Agile world — can help, but they often don’t uncover the reality of the actual system we’re building… just one similar to (and therefore potentially totally unlike) it.

The result is that reality rarely adheres to our estimates and plans.

So What?

It is probably high time that we admitted most software engineering isn’t actually engineering, in the strictest sense of the term. The implication is that our estimates and plans — and our ability to hit deadlines — shouldn’t be relied upon in the same way. Embracing this truth, company-wide, might help us.

Secondly, I’d say that Agile, yet again, has something to contribute: We already know that it helps to nail requirements by working in an iterative and evolutionary manner alongside a customer, but it also assists in handling complexity and planning, by removing strict estimates and by working iteratively.

Most Agile “stories” are planned in points, t-shirt sizes, and other schemes, in an attempt to get away from time-based estimates and instead plan based on the perceived complexity and past ability of the team to deliver similar stories. Agile’s iterative approach means we can adjust course as complexity and other problems appear, just as we adjust course by regularly showing early versions of a system to a customer.

One implication of accepting this is that software projects will be notoriously hard to plan against calendar deadlines. This means the wider company will need to work with software teams in a way that depends less upon deadlines, estimates and plans. Many organisations embrace Agile in their software teams, but not in the wider company. This creates the untenable combination of Agile software work practices, but rigid / calendar deadlines and deliveries to the wider business… and the largely unacknowledged fact that these two worlds seldom align.

Maybe it’s time we embraced the reality of this less-than-formal “engineering” activity of ours, owned up similarly to the reality of our less-than-dependable “estimates”, and relied more on ways to work with these truths, such as Agile? This may finally mean embracing Agile in a truly company-wide manner, rather than just within software teams.

For a Coherent Software Product… Draw More Diagrams

Pencil, By Juliancolton (Own work) [Public domain], via Wikimedia Commons

Unless you’re building it entirely on your own — and even sometimes then — for a unified and coherent software product, you need prominently-shared diagrams of what you intend to build.

Most software products are built by more than one person; more than one mind. Most are built by more than one team and usually across more than one skill or discipline. That’s many backgrounds, many viewpoints… but still one intended product in which they all need to be unified.

No-one would imagine coding an API without a spec, yet we often build systems without a diagram of the architecture, components, boundaries or intended flows. Retrospective diagrams often never happen, or happen way beyond the point it became painfully obvious they were needed. We over-rely on the verbal signs that we are aligned, without backing it up with a diagram.

Without a shared vision of the intended outcome, we’re likely to translate any purely verbal descriptions into different mental models, and onwards into different technical solutions… which then need to be aligned. “Oh I didn’t realise”, becomes all too common. This is particularly true in places where multiple disciplines overlap: Product owner, UX, front end, back end, operations, marketing, etc.

Everyone sees their piece of the puzzle, but diagrams confirm how they will all fit together. Without them, we introduce opportunities for hidden agendas and unnecessary extras to creep in. Human nature dictates that, where ambiguity exists, we tend to work silently to our own preferences rather than seeking a unified team answer. A diagram can provide that answer, or at least highlight the blank space in which it needs to go.

What to draw?

Whatever you find yourself discussing… but especially user interfaces, architecture, components, flows, and absolutely all inter-team or component boundaries.

How many drawings?

Just enough… but at least the highest levels of the product, then break down where more clarity is required and particularly where team boundaries exist.

Not too many though… or no-one will look at them. Find a happy balance.

What to draw with?

Informality, and a shared medium… Keep diagrams informal, because speed is important in encouraging capturing of info. Use a tool for more formal diagrams, but a tool that everyone can access; the person drawing the diagram will not necessarily be the only one changing it or adding to it later.

Where to draw them?

Visibly! Stuck away in a repository or a wiki where people rarely (or begrudgingly) go, diagrams are next to useless. Get copies of them on up on the walls too.

Update them!

An obsolete diagram is worse than no diagram. Think living diagrams!

When a change arises, update the diagram now, even if only with a scribbled annotation on a printout to be added to the original later. This is why informality often wins; there is no pristine copy that needs painstakingly updating.

Assign someone the action of updating the diagram, and make sure that they do.

Encourage diagrams

Make sure whiteboards are available, with pens (if you have to hunt, the moment will pass). Also make sure wall space is available, on which to stick diagrams that will live for longer.

Whiteboards aren’t just for visual disciplines; encourage engineers to draw components, flows and architecture too. Even short-lived diagrams, such as those drawn in one meeting, can unify people at a point in time. Get everyone comfortable with the idea of standing up with a whiteboard pen… and drawing. It should be a daily occurrence.

There should be a blank whiteboard usable by anyone who feels the need, at any time, and preferably within reach. This is a cheap resource, so over-provide. The improvement in communication, and in a coherent product, will be worth it.

Why Time as a Solopreneur Might Make You a Better Team Player

forwardTeamwork: Good. Lone work: Bad.

Or at least that’s how the stereotype drummed into us through education and our careers seems to go. We learn to focus our CVs and resumés on our team contributions, and to downplay the things we achieved entirely alone.

But what if the things we achieved on our own somehow made us more effective the next time we joined a team? What if a balance of both experiences made for better team players?

It’s my experience that teamwork between people who have never known an occasional lone accomplishment can suffer from a few blind spots: Such team members may have an inaccurate view of their contribution and how to tune it, having only ever seen their work as part of a larger deliverable. There’s a danger of feeling like a full four-wheel contributor, whilst actually running on three wheels and being unaware how much your contribution leans on the work of others. People who have never known personal wins can see a team project as their only chance to demonstrate their abilities, and may be over-invested in personal outcomes rather than team outcomes, safe in the knowledge that they can shine elsewhere or next time.

So what do I mean by lone accomplishments? Lone accomplishments could be a product (software, hardware, anything), a piece of art, training materials, blog posts, or a book. Anything you actually launch, publish or deliver to an audience, for which the entire deliverable depends on you, and for which you have no backup or other input to assist. Anything for which there is no scapegoat once you deliver it; it is yours, good and bad.

I’ve done this twice, with software products of my own. They were far from a commercial success, but the lessons learned were worth more than the time I spent and the salary I forfeited in those months. Since then, I’ve gone back to teamwork in startups, and really appreciated the perspective that my lone achievements have given me.

So what do lone accomplishers bring to a team? How can lone work strengthen and define you?

  • People with lone projects under their belt tend to have a more accurate view of their own strengths and weaknesses, and an appreciation for the roles others perform in a company.. because they’ve probably filled them at some point, albeit probably rather clumsily. Most former solopreneurs will have noticed how they couldn’t possibly fill all those roles alone… and will appreciate that it is only possible to cover them well in a team where a variety of skillsets are present.
    For me, as a software engineer, I now have a great appreciation of the other roles involved in startup life: Marketing, PR, design, QA, Ops, etc. Prior to undertaking my own projects, I probably had a dismissive or inaccurate view of those roles.
  • People with lone projects under their belt already have personal “wins” in the bag, so they don’t feel the need to act as if the next team project is their own pet project or chance to shine. This makes conversations more about a team win, rather than a personal win, less religious, more exploratory, and more practical.
  • People with lone projects under their belt tend to appreciate the difficulties of building something entirely alone, and therefore the need for others. The mature lesson from this is a willingness to delegate… and truly delegate, giving people a long leash and adjusting it appropriately. Knowing I can’t do it alone, but might be able to achieve it with others, makes me value the contribution of others and the need to delegate effectively.

Why is lone work a rarity? Aside from the widespread celebration of teamwork, we probably shy away from lone work because we know it will truly define or break us; there is no middle ground, and so we fear it. Through it, we will either discover, once and for all, that we can lean into a challenge on our own, or that we don’t (yet) have what it takes. Unlike teamwork, there is no scapegoat, no fallback, no-one else to blame. This is a hard reality to face.

But I’m convinced that every day we put off an attempt at lone work, is a day we never really understand what we are capable of; both alone, and in terms of our contribution to a team.

I’m convinced that, particularly in industries that involve building something (software, hardware, arts), we should exercise our lone capabilities on a regular basis, and at least once a year. It’s an ongoing eye opener on who we are, what we’re capable of, and therefore what we each bring to a team.

Far from being a red flag when hiring, lone work — when properly learned from — could be the thing that truly makes a strong team player.

Software Teams: One Vital Missing Question?

questionmarkBuilding software as a team may appear to be mainly about writing and releasing code. But it is underpinned by something much more fundamental: An ability to define and share information and complexity within the team.

It’s about making sure the ways in which we each model that complexity in our head match those of other team members. Unfortunately this doesn’t just happen magically, despite our best hopes and intentions.

If we aren’t all in possession of the relevant information and a similar view of the complexity we’re faced with, we’re forever building products that no-one wants (disconnected requirements and use cases), wasting time repeating development effort (wrong design / architecture / decisions), or building things that just don’t work (badly integrated, installed or deployed).

I’ve found that encouraging everyone to ask themselves one deceptively simple question, regularly and about almost everything they do, may provide the missing perspective that’s vital in keeping everyone on the same page: “Who else might need to know this?”  Yes, rather simplistic and, of-course, you have to act on the answer, but I think just remembering to ask that question about everything is the key change required.

What kinds of complexity and information are we needing to share as a team? For even the simplest of software projects:

  • Domain knowledge – Every business / sector has its own concepts. People arrive in the team with different backgrounds and therefore with a different need to familiarise with that domain knowledge. Without this, a fair portion of the thousands of little domain decisions an engineer needs to make each day will be inappropriate or wrong;
  • Terminology – For the domain and product, we’d better all be using the same words, right?! I’ve seen projects, where key terminology wasn’t nailed down early, go horribly wrong;
  • Research & Decisions – What we looked into (tech, architecture, anything), the decisions we made based upon our findings and the things we’ve learned. Recording these prevents us from revisiting prior decisions or mistakes unnecessarily, but also allows us to choose to revisit them, pick them apart and change our minds;
  • Architecture & Design – What we think we’re building, and how we’re breaking that down into components, infrastructure, etc;
  • Plans & Tasks – How we’ll build it, the individual chunks of work, sprint-by-sprint planning, agile boards… plus bugs as they arise;
  • Milestones – Points by which we’ll measure our progress. These should be in terms of usable chunks of the deliverable product itself, not just “sprint 1”, “sprint 2”;
  • Progress – Where we actually are versus the plan, and what else has happened on the way (because the plan always changes);
  • Product specifics – Usage, configuration, installation, etc. These arise throughout the project, and need capturing somewhere so the product is installable, releasable, usable, etc;
  • Product – The actual deliverable; probably many different versions of it and details of what went into each of them.

When it comes to sharing this complexity and information, most teams work on the basis that people will always ask what they need to know, at the time they discover the need. This is on-demand sharing. In an increasingly agile world, perhaps it feels like the right approach.

I’d assert that anyone missing a piece of information will often (though not always) ask what they know they need to know, but that may be incomplete. Even if they do in-fact ask, it will probably be for a very restricted fragment of information they know they are missing. What they will be without is context, background, and the top-down structure that would make the fragments more easily digestible by them. Context would also allow them to retain the information and to see the links — the bigger picture — rather than building their own mental picture iteratively from scratch, bottom-up, and perhaps by trial and error.

I’d say we should sharing all of this information much more keenly, and certainly before we think someone else may need it.

But isn’t keen sharing wasteful? Certainly not as wasteful as all those missed connections, delayed understanding, repeated work and the constant need to chase clarity. I’m not advocating writing lengthy documents, just capturing information, making it discoverable and keeping it up-to-date. Once captured, it can always be further structured at a later date.

For some of the information we need to share, there are already great tools out there, particularly task management and agile planning. But what are the best ways to share other information and complexity, and what are the pitfalls?

  • Online docs / wiki (lowest impact, longest-term retention, most transferable) – This is by far the best way to share to allow people to seek out information long-term and efficiently, but it comes with a health warning and an obligation: Wiki contents should be regularly reorganised for easiest navigation and digestion. They should also be updated, or marked as obsolete or, better still, archived when no-longer relevant. If a new hire needs the wiki explaining to them, it isn’t a wiki, it’s a dumping ground! The general rule is to encourage people to leave the wiki in a better state than when they found it. Besides the code base itself, the wiki represents shared team knowledge in its most accessible form;
  • Email – Whenever you email something, the need for which may go beyond a few days or a week, ask yourself the question “How will someone who isn’t on the receiving end of that email get the info?”. To retain the email benefit of highlighting the info to people right now, you may wish to email a link to a Wiki page instead, and this avoids the next new hire being unable to find it. Never email anything that doesn’t need drawing to the team’s attention now. If it simply needs to be available somewhere (a logical “somewhere”, preferably), put it on the wiki. Email inboxes are not a good repository for information;
  • Verbally / Whiteboard (highest impact, shortest-term retention, not transferable) – For the highest impact, nothing beats verbal communication; a stand-up, huddle round someone’s desk or a whiteboard, a meeting or presentation. Much of what was said may be lost beyond the occasion itself though, and it certainly isn’t available to anyone who wasn’t in the room (unless you recorded and uploaded it). Anything shared verbally or on a whiteboard should also be available on the wiki, either as a presentation / image, or by being absorbed into existing content after the meeting. Even just chronologically capturing whiteboard contents can help, and the structure can be rearranged later. Ask yourself if you’ve had to explain the same concept more than a few times. If so, it probably deserves to be documented somewhere.

Building software is about managing and sharing complexity. It may sound rather remedial, but encouraging people to regularly ask themselves “Who else might need to know this?” can affect a powerful change in the way a team works together and shares information. The answer usually prompts keen sharing and, if we also encourage the most appropriate methods and levels of sharing, it means that others will have access to the information when they need it, more often than not. Far from being cumbersome or time-consuming, keen sharing greases the wheels that lead to good teamwork… and perhaps even to better software.

The Best-Laid Plans of Task-Based Engineering

018-linkedinWhen faced with the challenge of building software of any considerable size, we rather sensibly tend to begin by breaking down the complexity and fleshing out a design or architecture. Then we break down the work involved in building that design into tasks or user “stories”.

Sometimes we group stories into wider “epics”, themes or stages, but generally we estimate, schedule, and conduct the work of building software at a task level: Tasks are what we assign, place on agile boards, and discuss our progress in terms of.

(I’ll refer to everything as “tasks” from here on, but I also mean user stories, and much of what I’m referring to also applies to bug fixes and other ticketed items)

Tasks are abstractions of the underlying engineering work required, and a task-based view of engineering makes the process of building complex software more understandable, manageable and distributable as a team. But it also tends to ignore certain truths about software complexity, tasks themselves and the ways we actually work on them, both on our own and together in teams. In-particular, the tools we currently use to manage task-based engineering are rather restrictive and cumbersome.

So, given that it seems we need task-based engineering, what is it about the way we approach it, and current task-based tools, that can hamper our efforts?

Tasks Abstract Away Complexity

A task is simply an abstraction of the work involved in building a software feature, a use case, a component or maybe just of fixing a bug. As with all abstractions, it necessarily hides detail that, for all but the simplest of tasks, may be discovered later when we actually work on it, despite our best efforts at planning time.

Sometimes, as the true complexity of the work involved in a task becomes clear, we realise that we should restructure the task to break it into more manageable chunks, and that sub-tasks should ideally replace some (but perhaps not all) of the original parent task. This discovery of structure and complexity may occur several times, to an arbitrary depth, and perhaps only after part of the original task has already been completed. We may even discover aspects of the original design, and the tasks slated to build it, are wrong.

Rather than implying poor planning, this is simply a part of the discovery process and the complexity inherent in building software, particularly in an agile manner where we’re trying not to pre-think too much up-front. We need to accommodate that inevitable change, reflect it in the tasks we are working on and in those slated for the future.

Most current task management tools aren’t great at letting us change anything more than very small numbers of tasks and task-based planning. They certainly don’t help us to visualise the results we end up with. In a sense, they leave us with quite brittle and sizeable task-based plans. With some persistence though, it is usually possible to reflect the required changes, though this may require time that isn’t available and we risk the task-based view drifting away from the engineering reality.

What can also be particularly tough with current tools is arbitrarily defining sub-tasks to any depth, as we discover complexity and break down a piece of work further. We may need to have parent and child tasks representing slightly overlapping and part-completed functionality, whilst being able to see estimates for all of that work. I’m not sure that current tools allow much of this without significant discipline in our use of them. In this sense, they are too brittle to allow us to reflect the engineering reality beyond our (now outdated) initial plan.

Tasks Need Surrounding Context

Our planning process envisioned the whole (a delivery, a completed system, or a bunch of features for a sprint), then broke it down into tasks. Quite often, rather a bewilderingly large number of them. The tasks somehow lose our view of the whole, and it is hard to determine whether those tasks, collectively, still make up a meaningful delivery or omit anything crucial.

It seems like we need to retain some visualisation or record of the structure by which we arrived at the individual tasks, so we can work backwards from the tasks to the whole. Even when working on an individual task, we often need to get enough surrounding context to work on it, as each task rarely describes that in sufficient detail.

Being able to see where tasks fit into the bigger picture is particularly necessary when our plan inevitably changes over time, to avoid us getting lost in a changing sea of tasks, the intended outcome of which we can no-longer visualise clearly.

Current task management tools don’t allow us to retain the more complex structure of groups of tasks. We are limited to “epics”, tags, simple sub-tasks and basic categorisation. We need better ways of visualising how the tasks fit together to form the whole that we’re working towards, allowing us to spot omissions and to track progress. After all, the collective structure of the tasks will probably be as complex as the structure of the overall system we’re building.

Tasks Need Collaboration (Sometimes)

No-matter how much we break them down, often a task requires unexpected collaboration with someone else (together, or a hand off then a hand back), integration with their work or reaching an agreement. That micro-collaboration is hard to model, but crucial to a task’s completion, though we often assume (perhaps hope?) it is irrelevant.

Whether or not one person “owns” the task and the responsibility for its completion, several people may need to do work on it and must find time in their workload. This is particularly true with user stories, as we don’t tend to break them down into chunks any smaller than a story we can define in business / product terms.

Current task management tools suggest a single assignee per task, or multiple assignees but with no record of individual effort required or how the collaboration will occur. We actually need to reflect a more complex sense of ownership and collaboration, even if most of the work will be done by one person with fleeting collaboration at some point. Failing to capture this in a basic form can be a cause of tasks being unexpectedly “blocked” during sprints, or the logical next steps towards their completion being unclear.

Tasks Have Dependencies (Sometimes)

Sometimes, a task depends on other tasks being completed (or part-completed), or a bunch of tasks need to happen in a certain logical order. Often a task that is scheduled to be completed this week or this sprint is blocked by an implied dependency that won’t be resolved anytime soon. We must model dependencies, or risk impacting our progress if we discover them late, or if many tasks get blocked on a key resource that we should have planned around earlier.

Current task management tools can model dependencies clumsily at the task level, but aren’t good at helping us to visualise them. This can often be another cause of tasks being blocked during sprints. They certainly aren’t good at modelling dependencies between groups of tasks, and we tend to have to visualise those ourselves: e.g. All tasks in Stage 2 depend on the tasks in Stage 1.

A visual representation of dependencies at a task and group level, and ability to move them around, would give us a more efficient view on how best to approach those tasks in a sensible order.

Task Completion Often Isn’t Clean

Often a task is almost completed, with a small piece of work to be finished, perhaps when unblocked by another person or after a dependency or decision is resolved. The inability to easily mark the task as partially complete, or to define the remaining work (perhaps in a sub-task), hampers visibility of the true state of the task, particularly on agile boards.

Creating a new sub-task can be cumbersome, and it needs adding into the plan or sprint, in correct relation to the original task, so it isn’t forgotten or confusing when seen in isolation. Often the administrative effort leads to us just leaving the original task open until the remaining work is completed.

Current task management tools assume task completion is clean and atomic, leading to tasks remaining “in progress” when they are often 90% complete and the tool should allow us to easily reflect that. This often hampers visibility of remaining work, particularly close to the end of a sprint.

So What?

Clearly, I’m pointing out some broad problems with simplistic views of task-based engineering here and suggesting some very general solutions. Not all of these problems will apply to a project but, particularly with larger teams, I’ve seen many of them and their impact is real. Some teams tackle these problems well as part of the way they work, but others clearly struggle with the realities of task-based engineering.

A quick summary:

  • Tasks are valuable abstractions for understanding and managing engineering complexity, but the methods and tools we currently use to manage them are rather rigid. We need to be able to visualise and change large numbers of tasks easily, see their overall status and visualise how they combine to form a planned delivery or feature;
  • We need to be able to break tasks down to any level, describe what overlaps and what’s already complete, so we can model our fluid discovery of complexity but retain an ability to estimate and gauge the remaining effort involved. Again, current tools and methods are too brittle;
  • We need to be able to model and visualise task and task group dependencies, where they matter (they don’t always), or even just record that entire swathes of tasks must take place in a certain loose order. Current tools make this too cumbersome;
  • We need to model the necessary collaboration within individual tasks, both simple and more complex, so we can be aware of it. Current tools are too cumbersome to make this effective or even worth the effort, and agile boards are too simplistic.
  • Most crucially… All of this must remain light, easy to visualise and easy to change. Current approaches and tools are way too cumbersome, and often work against us.

As they say, “Your mileage may vary”. So I’d be really interested in hearing how this ties into your own views of task-based software engineering, whether you encounter any of these (or other) problems, and if you think the kind of broad changes I’m suggesting might improve things.

One thing is certain: Task-based engineering is here to stay!