Hitting Software Deadlines: Essentials Only… then Upgrade

From “Deadline” can seem like a dirty word, in the software world. But without them, our work wouldn’t see the light of day, get an audience or have an impact. So despite their perceived unpleasantness, I’m all for them.

We tend to plan work towards a deadline in ways that ignore two inherent truths about software development, and therefore often miss them:

  • Complexity appears late: Despite our best efforts to surface it during planning and design activities. This is the reason plans go awry at the eleventh hour and deadlines that seemed achievable are inexplicably missed;
  • Completeness is hard to nail: Specifically, deadline-completeness. This means only the things that are strictly necessary for a viable release, launch or demo at the deadline. This is likely smaller than the list you feel you want to include, but adds crucial items that are usually not visible to a client, such as productionising, releasing, legal formalities, etc. Many of these are usually slotted in just-in-time before a deadline, or missed. Denial of the need to nail completeness, or discovery of further complexity here, can undermine a deadline.

Maybe we could try something along the following lines, to hit deadlines more often:

  • Find the Essentials for Deadline-Completeness: Begin by ruthlessly separating the work that must be completed for the deadline, but perhaps in a heavily-downgraded form.
    This might mean something releasable to a production environment (in a walled-off manner, minus a final trivial “reveal” on the deadline date), something that discharges legalities, and something that exercises the agreed use cases in a minimal but viable manner.
    Encourage people to write separate stories for bare bones deadline completeness, and then the rest.
    Also write meta stories for items required for completeness that would otherwise be forgotten due to not being visible to a client, such as deployments and productionising;
  • Hit deadline-completeness early: Complete the bare bones deadline-sensitive work as early as possible, and nothing else. In an Agile approach, this should be at least a sprint or two before the deadline itself. You will end up with a very minimal but released (though perhaps walled-off) system that would hit your deadline, but would perhaps disappoint you in a number of ways. Learn to accept this, because you won’t be revealing this version of your work and something much more crucial just happened that is worth celebrating: You already hit the deadline. You are already capable of delivering. So now, you can…
  • Upgrade, iteratively: Add in the work to upgrade that initial complete solution in tight iterations. Each of these should result in a system that is “better” in some way but still deadline-complete; remains releasable, still discharges legal obligations, still exercises the same use cases and perhaps some new and less crucial ones.
    Iterate on these upgrades, in a order of decreasing priority, until the deadline.

When the deadline comes, as it inevitably will, you are guaranteed to hit it in some viable and complete manner. There will always be upgrades you wish you could have included but they will be the least important ones and, much more crucially, you aren’t left with an incomplete solution that can’t released, demonstrated or used.

Approaching deadlines in this way means that complexity can be handled as it arises, without derailing the deadline. Hitting deadline-completeness in a minimal way, then upgrading, allows complexity to push out less-crucial upgrades if necessary. Yes, you don’t want it to… but the reality is it might need to. Post-completeness, there is room for the battle between complexity and upgrades to take place, however that plays out and without risking the deadline.

How do you persuade people to adopt this approach? By showing them that everything is still planned to be included, but that deadline completeness is prized first. Assure them that other upgrades, in decreasing priority order, will only be omitted if unforeseen complexities or problems arise. The maximum value possible is still going to be delivered. Nothing has changed but the approach to delivery and risk.

Isn’t this just Agile? If your approach to Agile (and there are many) includes a notion of deadline completeness, perhaps. But who really does Agile that way? After hitting the deadline early, the iterations are Agile-like. But this approach forces early inclusion of aspects that might be unseen to a customer in an Agile world — hence no stories — but which are crucial for the deadline, such as productionising.

Find the essentials and complete them early, then upgrade. By planning in a way that handles complexity like this, maybe we can let it do its worst — as it inevitably will — and still deliver to a deadline?

Kaizen, Innovation and the Entrepreneurial Life

10251565_830162623679537_14497031_nKaizen is a Japanese word meaning “good change”, but it has also been the basis of an approach to “continual improvement” that has been applied in many industries and as a way to drive personal change.

The idea is that incredibly small changes, made continually and consistently, tend to beat radical plans, proposals and initiatives because they work with, rather than against, human nature. Small changes bypass the brain’s tendency to fear and resist change. They also set us up to actually like the change and to want to accelerate it naturally.

It struck me recently that many of the words applied to startups and entrepreneurial pursuits are the polar opposites of Kaizen: They tend to imply “disruption” and sweeping changes carried through by radical innovations, actions and energy. But does this expectation of brave change on a huge level put our entrepreneurial dreams at odds with human nature? Are we hampering our own efforts by working against, rather than with, our brains?

What if, instead of trying to find one radical way to “disrupt” an industry, we tried to notice small things that are true about that industry, make small changes to improve those things, and to use that momentum to gradually see the bigger possibilities? What if we tried to notice small ways in which we could improve someone’s life with a product or service, and build on that? What if we tried to innovate by making a small but meaningful change to an existing product or technology and see that as the basis for a bigger change?

And when it comes down to ourselves, as entrepreneurs… What if, rather than trying to affect the various changes in character, lifestyle and approach required to become entrepreneurial overnight — as if a butterfly-like metamorphosis in human behaviour is even possible — we tried to make small changes to our thinking, attitudes and behaviour to become more entrepreneurial over time? Kaizen techniques for personal change highlight the benefits of ludicrously small changes at first, to beat resistance. Those changes tend to snowball into genuine change driven by the subtle rewiring of our brains caused by adopting the smaller changes. There is certainly less chance of us reverting to our old behaviours if we affect change gradually, and it tends to be longer-lasting.

Of-course, all this talk of small changes goes against the grain of an entrepreneurial world where we value disruption and innovation. But what if, rather than distracting us, making these small changes increased the chances that we’d actually spot the bigger and more radical possibilities? It’s certainly easier to see what’s possible when you’re moving in the right direction, rather than standing still waiting for inspiration to strike.

In our quest for disruption, bigger and better, maybe we could do with realising that small and continuous change could be the most effective way there, albeit perhaps rather a counterintuitive one.

(Image: (c) 2014, Ian Cackett)

Looking at New Product Ideas the Wrong Way

idea-comfort-zoneThe moment you’ve had a new idea for a product, something dangerous happens… You start viewing the world from the perspective of that idea; validating the idea against the world, the people in it, and the opportunities / benefits it might present to them.

Sounds right doesn’t it? Intuitively, yes. But it’s the wrong way round!

What we need to do after having a new product idea is to keep validating it from the perspective of the outside world: Keep finding scenarios into which the idea might fit, and assess it in terms of its true benefits. Keep starting from the point of view of different sets of people in the world, or different potential use cases… and look at the idea from that viewpoint. Not the other way around.

The idea must not become the lens through which we see the world. If it does, we’re blinding ourselves to the reality and we risk building something that is of little use or limited appeal.

It’s counter-intuitive because, once we’ve had the idea, it is naturally our obsession. If the idea is something we feel we could build as a product, we can get blinded to the fact that there are a billion things we could build as products… but a much smaller number of things that would truly benefit the world, and perhaps even manage to be commercial products.

I think it’s a subtle but powerful switch in perspective.

 

The Subtleties of Validating Ideas

I guess those of us following a “lean” approach to developing software product ideas talk a great deal about how we “validate”  them.

The usual wisdom is that you expose your idea — landing page or, better still, your MVP — to the world, preferably in its leanest  form possible, perform some sort of marketing activity to draw attention to it (advertising, social media, word-of-mouth), and then measure what the interest is, usually in terms of sign ups and usage versus number of eyeballs. This is intended to give you a very rough idea of the size of the market, and perhaps an informal sense of how much it might take (and cost) to “acquire” each user.

In general terms, I believe this is spot-on. But there is a great deal of idea-specific subtlety that I think you have to gauge each time you do this. I don’t think anyone can distil or teach it and I guess this is where much of the benefit of trying and rejecting ideas comes from. There really is no substitute for actually doing it.

With some (possibly almost all) ideas, validation probably requires that you go out there and somewhat coerce  a few people into becoming users of the product. This is what Paul Graham refers to in “Doing Things That Don’t Scale“. It feels unnatural for techies like me, but it’s probably an essential part of the process. There simply is no way to validate the idea itself, until an MVP is built based upon it, and even then passive validation (“It’s here, anyone interested?”) usually isn’t enough to pull anyone out of their busy lives for long enough to take a real look at what you’re offering.

Potential users just don’t often have a gut sense of whether your product is attractive to them, until they’re sat in-front of it, using it… at which point, the lightbulb above their head will either light, or it won’t. Longer-term, they might help to spread the news. But initially, you may have to wire up the light bulb yourself.

Very few products leap straight off the landing page and on to the “sign up” page. And those that do are possibly the result of a process of maturing from a much more manual marketing approach, and no-doubt took many, many iterations to get to the point where they appear to market themselves effortlessly.

I think it’s a balance though between how lean  an offering you initially run with, and how hard  you push it at people. It really is a question of learning to judge when you’ve exposed or pushed the idea way too little, versus bullying people and trying to magic up a market where none exists. Judging this is quite an art form.

Scaling Software Products, in a Nutshell: Why, When and How

Startups building software products tend to think about scalability as an after thought, quite understandably. There are far more pressing things to focus on first, like identifying a product that solves a problem, for a paying and addressable market, delivering value to that market, and then collecting revenue from them for doing it. All of that must take centre stage, at least initially, particularly if you’re following a lean approach to getting started and are launching to validate an idea with an MVP. (Hint: Too much thought about scalability probably violates the “M” for minimum).

At an early point in the lifecycle of a product however, someone inevitably asks “Does it scale?” — Meaning, will the product continue to work if we have 10 times, or 100 times, or 1,000 times the current number of simultaneous users? And, more interestingly… at what point will it start to fail?

Not surprisingly, the initial answer is often “I don’t know”. And this is a concern.

So here, in a digestible nugget, is what I’ve learned about the huge topic of Software Scalability in 20 years as a software engineer, so-far (I’m sure I have much more to learn). I’ve focussed mainly on server-side scalability of web applications, and particularly those deployed on Amazon Web Services (AWS) or Heroku (or similar) here, as that’s my area of expertise and close to what most startups are looking to scale. But many of the ideas here also apply to other types of software product.

SO WHY EVEN THINK ABOUT SCALABILITY?

In two words: Customer Experience.

If your customers have a bad experience when using your product — a slow response, a timeout, an error message — they will, in all likelihood, look elsewhere. With poor scalability, you can literally lose customers, often before you’ve gained them.

The basic — and often overlooked — software engineering fact that drives the need to consider scalability is that what works well in your Development environment — usually on your laptop — and seems to do similarly well when initially deployed, perhaps to AWS or Heroku with very little traffic — and by traffic, I mean users — may not work so well when you publish your first press release, get mentioned on Hacker News, or in the days after you give your first kick-ass presentation and everyone flocks to your site to find out more. These are all happy events… but only if your product scales to meet the increased demand, which naturally requires increased resources.

Being able to offer every visiting user a reasonably good experience of your product  is as key to success as figuring out what your product should be. A great product that is slow to respond is unappealing at best. A product that is hidden behind a timeout message is essentially worthless.

WHEN TO SCALE?

Ideally, just before you need to.

Metrics – Just like forecasting the weather, gathering and using metrics is the only real way to tell how your product is doing, whether it is handling the current load and to predict where the limit and the degradation in user experience might begin to occur.

Instrumenting your product code (by capturing your own timing stats, perhaps using Aspect-Oriented Programming or a third-party library), measuring throughput (hits/sec, average response time, etc) and then capturing and aggregating those metrics using a product such as New Relic, or building your own performance dashboard, are all great ways to expose the current performance  of your product in near real time.

Spare Capacity – Metrics only show how your product is currently doing. What you’re more interested in is the spare capacity: How much more traffic you can handle before problems start to occur. If you are comfortably handling 1,000 requests/minute right now with a decent average response time and customer experience, but you aren’t sure whether the 1,001st request will start to cause timeouts, then you know nothing.

This is where Performance Testing, in a separate but (crucially) production-like  environment, lets you figure out by experimentation where the limits of your system lie. Regularly performance testing and benchmarking  your product, graphing the results and determining at what point a single server begins to give an unacceptably-degraded user experience, lets you forecast the capacity and breaking point of multiple servers handling the load between them. This, in-turn, gives you a picture of the spare capacity you currently have. Even better, baking all of this into your product metrics, and reporting a percentage capacity remaining, lets you know how close to the need to scale you currently are.

It is worth saying that any such notion of capacity needs regularly reviewing, as changes to your product, the hardware it is deployed on, or other random factors can change capacity in unforeseen ways.

Automation – Metrics alone aren’t enough. It’s no use having great metrics if, at 3am when your product gets mentioned on Hacker News, you don’t notice the hits escalating, average response times dropping and the number of timeouts going through the roof. Automation and scaling policies  allow you to define conditions under which you’d like to scale up, even when you aren’t around to make the decision yourself. Unless you plan on watching them 24/7, or getting out of bed to answer the Pager Duty call, your metrics, and in-turn their notion of dwindling spare capacity, must trigger some sort of scaling action.

Cycles – When to scale is a never-ending “when”, and it is often cyclical: As well as wanting to scale up to meet increasing demand, you might want to scale back down again  in quiet times to avoid wasting resources. Any scaling policy should also define how much spare capacity is too much, if only so your AWS bill doesn’t hit the roof and wipe out your profit.

Metrics, coupled with automation tell you when to scale, but not how.

HOW TO SCALE

Scaling UP

scale-up

Scaling up (often called Vertical Scaling) involves running your product on a “bigger” server, meaning a combination of more processing power (CPUs / cores), and/or more memory. How much more of each you require is determined by which limits you are hitting. If your product is CPU-bound when under load, you’ll need more processing power. If it hits memory limits, then add more memory. Usually, a mix of both is required to scale up effectively.

Scaling up is easy to accomplish because it requires no architectural changes to your software. You just deploy your product to a larger instance (on AWS or Heroku), or buy a bigger server if you are hosting the product yourself. The only problem is that it needs to be a pre-planned affair, unless you can seamlessly upgrade your instance as the need for additional resource grows.

Scaling up should, however, be your first move. You should be running your product on a box that will — on its own — meet projected capacity for a while without needing to scale further.

Scaling OUT

scale-out

Your next move is to scale out (often called Horizontal Scaling). This involves distributing traffic across multiple servers. Hosting providers such as AWS provide load balancing technology to achieve this, spreading the load across however many servers you currently have running.

Scaling out introduces some architectural challenges but, unlike scaling up, it can be adaptive: You just add additional resources to meet demand, and remove them again when you no-longer need that capacity (though there are challenges in scaling down too; see later).

One of the main challenges with scaling out is that the state  of your application is no-longer held in memory within one server, and is distributed between them. If you are caching certain information, for performance reasons, one server’s view of that information may be out-of-date if a change is applied via another server. Simple solutions involve techniques such as “sticky sessions”, whereby individual users are always load balanced to the same server instance, whilst it is running. AWS offer this as an easily-configurable option. But the state of your application may not be easily subdivided by user. You may need to use your database as a place to update shared data transactionally, such that race conditions don’t cause you to lose or mis-calculate data.

Other things to consider are the use of a message bus  (such as RabbitMQ or Amazon’s SQS), such that information may be shared between servers; even something as simple as telling all instances when to invalidate specific data in their cache, can be easy to implement via messaging.

The architectural challenges introduced by scaling out are beyond the scope of this article, but the benefits of tackling them and being able to add spare capacity whenever you want are certainly worth it.

Incidentally, I mentioned scaling down being a challenge. You can’t simply remove spare capacity from behind a load balancer, because the instances may be in the process of handling a request from a user, resulting in an error or timeout. AWS now offer “connection draining”, such that instances that are about to be terminated are given a configured period of time to finish handling existing requests, but no new requests are routed to them. This makes scaling down as effortless as scaling up, if correctly configured.

Heterogeneous Environments

heterogeneous

So-far, both methods of scaling have involved either resizing or duplicating servers running the same software, and performing the same role. However, certain activities performed by your product may form a larger part of its performance bottleneck, such as expensive computations.  It may make sense to scale portions of your product separately from others, such that the spare capacity and scalability of one aspect may be controlled separately from that of another aspect.

This might be most easily achieved by separating the product into Services, and scaling the number of instances of each of those services separately to meet demand. Some services may share the same process, but others may run stand-alone, perhaps connected to the other services via a message bus, socket connection, or just by virtue of sharing the same database.

If certain services perform computationally-costly activities, you could assign them to compute-optimised instances and scale the number of instances based upon the size of the queue of waiting work, or the average turnaround time for tasks, or another metric you’d like to optimise.

Separating concerns in this manner becomes more cost-effective as the overall size of a product deployment increases. There is little point, initially, in doing this with a small deployment and everything should probably run in the same process, or at least one of several identical processes, at first.

DELAYING THE NEED TO SCALE

Sometimes, it is entirely possible to delay the need to scale. In web applications, if certain computations — particularly those involving just aggregation or simple processing before presentation — can be offloaded to the browser, without increasing bandwidth or affecting user experience, this can save server resources and means the server-side cost per user  is reduced. This, in-turn, delays or reduces the need to scale.

Where activities are not time-sensitive, scheduling  when they occur to coincide with times of lower traffic, can also reduce to server resources required at any given time, and delay (or entirely avoid) the need to scale.

SUMMARY

Software Scalability is clearly a huge topic. I hope this is a useful overview, particularly if you don’t come from a software background, and that it leads you to delve deeper into specific areas. At the very least, you might start building metrics into your product and use them to figure out when Scalability is likely to become a more pressing concern for you.

Startup Founder Stories are Signposts to Success, not Prescriptions for it

changing-trajectoryThe stories of successful startup founders collectively point at — but don’t reliably prescribe — the ingredients for our own startup success.

There’s a particular genre of books in the startup world, which mainly consist of the “How I Made It”  accounts of successful startup founders. One per chapter, and usually not much else. I find these books interesting to a point, but it’s like reading an account of how someone created a great painting… You’re left feeling you’d much rather to go and create your own painting than read about theirs.

There is one fundamental flaw in seeing these accounts as anything other than inspiration and signposts: They are accounts of how someone made it with that  product, in that  market, with those  co-founders and — most crucially — at that  specific point in time. Almost all of those ingredients will be fundamentally different for our own startup story. Even just the passage of a few months of time changes things immensely.

Sure, there’s much to be taken away in terms of how and why they made specific decisions then. But the decisions themselves are less relevant than their thought process  at the time. When making our own decisions, the factors to be taken into account will be different and the best decision may be the opposite one from theirs. The specifics of their decisions may in-fact blind us, when what we should be trying to distil are the principles of good business  and good startup commonsense  behind them.

So I read these accounts and try to figure out, in aggregate, the good advice and principles within them. I’m less interested in the specifics now, because they are a matter of history. It is far more interesting to figure out what I can do that might influence my own startup outcomes in the future. Historic accounts help, but only to a point.

 

Remembering To Dream Big, Again

S025-bwFrom the age when I could first put the pieces together, I built impractical things in Lego, and sometimes in Meccano. I used to dream of having enough Lego to build a castle, or a Meccano crane as tall as the house. There were no limits to my dreams, other than the size of my Lego set.

When I got into Computing I realised that, in a virtual world, there was no limit to what I could build to improve, serve and augment the real world outside. It was the ultimate toolkit. That’s how I found my passion, and my career. I count myself as immensely lucky.

Sometimes though, I wonder if I am honouring those early dreams, or whether I let them fade along the way. Have a Zone 1-3 travel card, a monthly pay slip and someone else’s dreams made me forget that being alive necessitates an element of impracticality, of thinking big… no, bigger!  Of planning things that might initially seem beyond me, then stretching to reach them. Not everything worth pursuing can be neatly described in a career plan, a yearly goal or an appraisal.

It doesn’t matter how many hundreds of thousands of lines of great Java code I’ve written, if I’ve not yet written the thing that reignites those early dreams.

I feel privileged to know some immensely passionate and brave people — entrepreneurs, artists and professionals —  some of whom are pursuing their dreams and making them real seven days a week, not just in “spare” moments snatched from a busy work schedule. I think it’s no coincidence that I’ve stayed in touch with these people. Like them, I also pursued my own projects full-time for a 2-year period recently, got a real taste of what’s possible when I focus, and realised how tantalisingly few steps there are between a software engineer like me and viable, commercial ideas.

I have, in Computing, the ultimate Meccano kit, the biggest box of Lego. It’s no wonder I regularly think, “Am I dreaming big enough?”

 

 

Making “Impossible” Post-Launch Product Changes Possible

Image: (c) 2014 Ian Cackett

Image: (c) 2014 Ian Cackett

Before launching a software product or app, any tech change is possible.

One day, you might change the database (MySQL seemed a bad choice anyway, right?). Another day the architecture. Perhaps the front-end templating language turned out to be clunky, so you switch to a better one. Maybe you even rewrite the whole thing in an entirely different programming language (because your mate down the pub said the one you picked originally sucked… and other fine reasons!)

During these pre-launch days, anything seems — and indeed is — possible. If something breaks, you just fix it and move on. No-one gets hurt.

But these heady days come to an end… After launch, certain changes become tough, scary or downright “impossible”. With existing users, data and promises of uptime / availability, it no longer seems possible to make certain changes. You are locked in by the burden of live usage. Or so it seems.

You see, I believe that it is always possible to make brave, far-reaching and deep tech / software changes, despite having already launched the product, but it requires three somewhat uncommon things: A reason, a plan and the willingness to execute that plan to completion. Miss one of these things out, and chaos is almost guaranteed.

Reasons – Before even starting, is there a reason for this change? Pre-launch, changes can be made on a whim. But as the product is now live and this will take some considerable effort, reasons are important.

Does the change improve the product in some way? Perhaps its performance or even just its maintainability going forward? Perhaps it allows the business, or users, to do something with the product that they can’t currently do? Perhaps it opens up the possibility for other desirable change down the line?

If there aren’t tangible reasons for a change, don’t do it or schedule it for another time… or for your next product, if this is just a personal preference for a change or a desire to follow a trend. After all, the product and its users matter more than how you personally feel about the code.

Plan the Destination – Ask yourself whether the proposed change would work, both trivially and at scale. This is the time to try the change in a test environment, but as-yet with no eye for how  you’ll get there and absolutely no code committed that will affect production.

Does the new database improve performance? Does the new templating language actually make things simpler? Does it work under current production load? (Folks usually forget that last one, but it matters and you may need to build an environment that closely mimics production to figure this out).

In short, would the change justify the reasons for making it and actually work in production? If not, you can ditch the whole plan right now. If it does… move on and plan it further.

Plan the Journey – How would you migrate? Usually, this means existing data, but it also means dependencies between services and components.

If components depend upon one another, plan a step-by-step migration such that all components have what they need (data and other services) at each stage. If the system will be in use as the plan proceeds, how do you ensure that no data is lost? Perhaps users need to be prevented from certain actions or use of particular features at key times to ensure this? Perhaps they need locking out completely during a published maintenance window?

The plan needs to be rehearse-able and repeatable, so write scripts for data migration and other tasks that can be automated. The plan also needs to be written down and reviewed. This isn’t the time for napkins or the back of an envelope.

If the plan seems to work, rehearse it again once more. Sometimes you do unexpected things on a first run through that invalidate the plan in unforeseen ways, so run through it again cleanly one last time.

Drawing out a plan at this stage allows you to see whether there is a path to the destination. Can you get there with everything intact and no outages?

Aside from planning how this will take place in production, how will you do it in development and test environments? When does the code get committed? When do other developers need to make the change? When does that get promoted to various test and staging environments? This may all need to be over a protracted period, so making a code change that breaks another developer’s ability to support (and release to) production isn’t allowable.

Be willing to think about all of these in your plan, or don’t attempt such a change… you’re simply not methodical enough.

Plan the Timing – When would the production switch-over occur? When is data going to be migrated? (You wrote and rehearsed a script for this already, right?) Does it need to take place during a quiet time in production, a planned maintenance window, or can it proceed seamlessly at any time?

Plan for Failures – What are the contingency plans, and could you revert if the unthinkable happened and the journey became unfinishable at any stage? You might discover way too late the reasons that make the change impossible. What happens if you lose some data? What happens if customers can’t use the app for a period?

Plan the Impact – Does this program of change affect any others? Any parallel development? Any business users?

Whilst you’re planning this  program of change, think about other  activities also going on… the timing of the change may affect them too. Is someone else developing code on a branch that needs merging with yours? Is someone else planning a brave change that clashes with yours? Can you align them both, or perhaps combine them?

Plan for Stability – After the change occurs in production you can just forget it, right? Wrong! What you now have is a radically changed product in production. There needs to be a planned period, post-change, in which you are on-guard for unforeseen problems. Memory leaks, race conditions that didn’t become apparent in load testing, etc.

This is a reason why there should be a pause before other sweeping changes are layered on top. You need to see the system working for a period once everything is complete, and leave a window in which critical fixes can easily be rolled out without clashing with other changes. You need to see stability, after the change.

Execution – This is probably the one everyone forgets. Having planned the change, executing it requires one more uncommon thing: Rigour. You need to stick to the plan, at all stages, and ask regularly “Where are we on the plan?”

If the data migration will take place over 4 stages, make sure everyone remains aware which stage you’re at. If the upgrade takes place overnight, keep everyone informed of its progress and outcome. Did it work? Did you revert? Are there issues that people need to be aware of?

This isn’t the time to change the plan unless unforeseen circumstances dictate that. Whilst execution may seem boring, failure to be rigorous will undo all of the planning that went before, and could lead you into a mess of production problems.

With a willingness to plan and execute, and some rigour in both, I suspect that most “impossible” post-launch changes with clear benefits are in fact quite achievable. Their cost and risk is greater than pre-launch, but I would argue that sometimes these brave changes are worth the effort, and that entertaining them can help to avoid product stagnation or bigger problems long-term.

I think this is a more mature version of Facebook’s “move fast and break things”. Perhaps “plan, and don’t break things”. All change is possible, even post-launch.

The Importance of Risking being Mediocre

averagelYesterday, one of my oldest school friends, an artist, said something that made me pause and think about what we risk by pursuing our own projects, startups, goals or art.

Far worse than the often talked about risk of failure, or the similarly discussed and dreamed of success, is the middle ground.

“What if I’m mediocre?!”, she said.

Rather than being something that most people fear openly, I suspect this may still be the reason why a great many people never get started and never produce their own work. Meaning, work which we publish as our own, to which we 100% attribute our own name, not that of an employer, a course or a group. Work for which we own all success, failure or other outcome.

It’s a myth that we can somehow learn enough to avoid being mediocre at some point, during a safe preparatory period, and then go out into the world and pursue success directly. And yet, I suspect most never get beyond the preparation stage, because mediocrity is always a risk.

In “Clarity”, Jamie Smart says “Discover your path by walking it”. I think we actually need to begin and learn en-route. Create and publish or share our work. Naturally, this means going through any mediocre stage in full view of people who may criticise us, but I suspect it’s the only way; through mediocrity and beyond. There is no way to avoid it, and perhaps it’s better to embrace it as an essential aspect of success.

This is why I began blogging and, as a software engineer, why I launched two apps of my own: They are learning experiences and, each time, I get better. The only way to something bigger and better is by doing real work and putting it out there.

It is also work realising that, by putting our work out there… we are already well ahead of the pack, because most never will.

I hope you enjoyed this mediocre blog post 😉

7 Limiting Things Software Developers Need to Stop Saying to Ourselves

Going it alone

(c) 1993 Ian Cackett

As a software developer for the past 20+ years, I know that I have at times restricted my own view of what’s possible in my career. These were my own personal limiting beliefs  and things I was telling myself, but I believe most of them are quite common.

I’ve listed them here in order, smallest to largest. I suspect most people reading this will have conquered them to a point, and the remainder probably define where progress might be productive and possible. I am by no means done with this list myself, and my current focus is the last two.

How many of the following are familiar to you?

“I am just a language X developer” – We all start off with, or predominantly use, a specific skill set. Over time, it defines us within a company and in the wider recruitment market. Switching to other skills seems impossible, but it is entirely within our control.

If I hear another person say “But I need training”, or ask me how I learned Java or JavaScript as if I was somehow handed the opportunity on a plate, I will scream. Order books online, read tutorials, do programming exercises, teach yourself, build a working prototype of something in that new language, in your own time. You  drive your career progression, not your employer.

This doesn’t just apply to programming languages either. We are all capable of become a more well-rounded software engineer. If you didn’t study Computer Science or Software Engineering, read a few books or blogs and figure out a reading list to round out your knowledge. Learn about good design, architecture and listen to how others built systems that work well and scale. There is no hidden magic here, and anyone with a will to learn can reproduce the same great results as others.

“I only build the product” – Product direction and input tends to come from non-tech departments: Sales, Marketing, Product Management. It can feel like we have no input when, in-fact, we have a great deal to offer.

Listen and contribute to the business. Learn how to describe technical problems in non-tech language, and to translate business problems back into technical solutions. An engineer with these abilities has a great deal to offer the wider company. Beyond being a full-stack developer, extend that stack to some include business knowledge.

“X isn’t possible in this company” – Courageous programs of technical and business change can be accomplished, and you can lead them. If you learn to act as the rock around which they are planned and executed, they are possible.

Migrating a data centre to a new architecture, removing old redundant database tables, standardising the non-standard and even redeveloping the product from scratch, if justified, are all possible. I see so many of these remaining as long-term obstacles, and yet these are just the kinds of challenges that allow an engineer to prove themselves to a business.

The secret is to propose a plan that will actually work  — perhaps test it in a staging environment to prove this — and then act as the rock around which that plan executes. Split it into multiple stages, and define what must happen at each stage before moving on to the next. How will you ensure no data is lost? How will you be sure the business has moved along with the technical aspects correctly? What are the contingency and reversion plans? Think of the impact on the whole business, not just the bits that are relevant to you. Write it up as a proposal, in both business and technical language… but then expect to be asked to deliver it.

“I have nothing to teach others” – Stand up and teach the things you know. Risk being seen as audacious. But also listen and learn.

If you fear giving presentations, get over that fear. This has often been described as the single most limiting factor in most people’s careers. If you learned a new skill, be willing to discuss and teach it. Not willing to do that? Then don’t kid yourself that anything but fear is the reason. Fear is a self-imposed career obstacle.

This all comes with an expectation of pragmatism  though too: Don’t become the techie who always evangelises about the new language or framework, without an eye on practicality and the actual value to the business. If Scala will really benefit the business, learn it, propose it and be willing to teach it to people. If it’s just your latest favourite language, or the 12th one you’ve proposed this year, then keep it to your spare time instead. At least for now, anyway.

“I can’t talk to clients” – There is a stereotypical image of the techie whom business folks fear putting in front of a client. Don’t be that guy.

Learn to talk in the client’s own language, and to listen. The only thing that matters is translating their requirements into technical implementations, and demonstrating to them (again, in their language) what a product can, or could, do for them. Talking on their terms is key.

It won’t take much client exposure before you learn what to say, and what not to say. If you aren’t getting that exposure, ask for it. Sit in on a Sales call, ask to go on a client visit with someone… and then listen. You may be amazed by how many questions come up that you’re the perfect person to answer. And that’s your intro. You will most likely be asked to contribute again.

“I couldn’t build my own commercial product” – You already build some or all of a product for your employer, so what’s the difference in building it for yourself?

What we tend to mean is that we need someone else to constrain and shape the commercial context. But really, what is more potent than someone with both business and technical knowledge? Someone who can figure out that a client wants, and then build it or lead a technical team in doing that? You can, and should, add that commercial awareness.

I’ve begun reading “The Personal MBA” (Josh Kaufman), in an attempt to round out my own business and non-tech skills. It is my assertion that every technical person is a tantalisingly-small number of steps away from being sufficiently commercially aware that they can not only build a product, but figure out what to build too.

“I couldn’t run my own company” – If you can build a commercial product, what’s to stop you starting a company of your own to sell it? After all, a startup is simply a bunch of folks (or even one) who realised that they could do just that.

Again, the Personal MBA helps here, and an awareness of which areas you don’t currently know much about (your blind spots), plus a willingness to go and learn enough about each of those areas. Don’t deny the need for some marketing, sales, financial knowledge; They are all  crucial. Just admit what you don’t know, plan to learn some, and then realise you’ll need to wear each of those hats periodically to build (or at least kick-start) a well-rounded venture.

For me, awareness of what makes a product commercial, and how to run a company, are the two aspects that I’m still trying to learn. They have been pivotal in my own limiting beliefs in the past, and I’m beginning to see possibilities where once I only saw barriers and obstacles to my career.