Remembering to Sharpen The Saw

If you’re a software engineer, like me, spending most of your time working on projects and ideas, you probably only end up using the specific skills and knowledge required by those projects. It’s likely that you rarely get a chance to review what you know in other areas or to look into new topics, programming languages and technology.

I just spent 3 months on a rewrite of Changes That Stick, which focussed heavily on JavaScript, a lot of HTML5, some CSS (though let’s just say I’m many miles from “guru” level 🙂 ) and quite a bit of server-side Java (my main focus for 14+ years before that). What I haven’t done since last year, however, is take the time to review the current state of the Java world and fill in any gaps in my knowledge there, particularly Java7 (SE and EE) topics. Java seems to move on at a fair rate.

With this in mind, I’ve been on a (Kindle) book buying spree, and will be spending a few hours each day filling in those gaps, whilst working out ways to put the new knowledge to use… because nothing solidifies a new topic like actual use.

But I’m also conscious that I should be looking at new areas. Earlier this year, that included JavaScript; a language that I’d known a little about for over 10 years, but never deeply immersed myself in until then. I suspect, like many, I originally learned about it in haste and through a kind of reluctant necessity, and so I needed to go back and learn it for a reason. I ended up doing just that, and even learned some Node.js, then re-wrote my UI framework (Kaleidoscope) to be entirely browser-based rather than server-driven.

So because that worked so well last time, it’s time to pick a new topic again and I’m currently torn between another language supported by the JVM (which would seem logical), or something entirely new. I haven’t decided yet.

If you don’t “sharpen the saw” (as Stephen Covey calls it in “The 7 Habits of Highly Effective People”), you can end up reaching a point where your skill base has dated quite severely, through only looking in rather a blinkered manner at the next project, and the next project, etc. You have to pause occasionally and figure out how to maintain and add to the tools you have at your disposal.

It’s hard to find the time, but I realise it may also help me to come up with new project ideas. Often when you aren’t looking directly for new ideas, connections are made and they seem to pop up. Yet another reason to sharpen the saw occasionally.

Expanding our Startup Idea Comfort Zone

idea-comfort-zoneEvery one of us places new startup ideas that we hear or think up somewhere along a personal continuum from “That’ll never work” to “Could be the next big thing”. Instinctively, we make that call based upon our experience and our view of the world. No two people’s views on this will ever be entirely aligned.

To an extent, we limit ourselves by ruling out ideas that could potentially work. Similarly, we back other ideas that probably don’t stand a chance of ever getting built, let alone finding a market. In that sense, our view of the potential of each new idea is skewed versus reality but, as we’re essentially trying to predict the future, the only way to know that is with hindsight after an idea has either failed or worked.

The implication of this is that we need to run with a few of those ideas, see where they go and get a chance to properly qualify them, in order to get the opportunity for that hindsight and that learning to occur. I doubt that judging the success or failure of other people’s ideas really causes the same degree of learning. You have to do it yourself, and risk getting burned.

But here’s the crucial thing… If we only ever follow the ideas we already suspect are potential successes, we either get to confirm that they do indeed work, or we learn a new way in which our view of the world was over-optimistic. Our idea comfort zone will stay the same, or shrink.

So we also need to follow a few of the ideas that don’t sound like potential winners; the ones that might work, but we aren’t too sure. We need to learn to place a few bets just outside of our comfort zone, just beyond the areas we understand 100%, and in places where markets don’t necessarily exist yet… but could. In that way, we’ll either confirm the idea won’t work and strengthen our original assertions for that, or we learn that it is in-fact feasible and our idea comfort zone expands.

Why is this crucial? Because I suspect most of the exciting, new and worthy possibilities for startups lie outside the comfort zone of the average entrepreneur. They lie in unexplored or early markets, in unexpected offshoots from existing ideas, and have business models that may take a degree of exploration before they are understood. I suspect they don’t lie in mimicking other ideas, or tweaking and repackaging existing business models.

And how do we explore? With prototypes, MVPs, and a lean approach. When ideas have a degree of uncertainty, the quicker they can be qualified, the better. A lean approach lets us place more bets outside our comfort zone, essentially spreading those bets but shortening the time taken to qualify each of them. This is definitely not the place for lengthy stealth mode development cycles followed by grand unveilings.

Perhaps in the hunt for startup ideas, as in life, time spent outside of our comfort zone may be more rewarding than time spent within it.

Landing Pages Come In Many Flavours (and my search for the right one)

Getting the landing page for a new app or product just right is clearly crucial to its success, particularly when qualifying an early idea. After all, it is an inevitable fact that most people hitting that page will only ever see that page, before deciding to go elsewhere instead. Sad, but true.

A decent landing page is pivotal in converting users, getting them interested and getting them to sign-up. Beyond sign-up, we have many more options and opportunities to retain them. But before sign-up, one glance at the landing page may be our only chance. In a world of decreasing attention spans and competing or overlapping products, it is even more crucial to get this right.

I’ve been experimenting with various landing page options for Changes That Stick, with differing degrees of success. I’ve tracked behaviour on that page and, though sometimes interpreting the numbers is tough, I think I’m finding out what works and what doesn’t (mainly the latter, so-far!) Here are my early findings. They may be useful to others, and I’d certainly be interested in hearing what worked (and what didn’t) for you.

Traditional big blurb approach – To be honest, my first stab at a landing page was rather ham-fisted. It consisted of way too much text, a rolling slideshow of app images, scrolling tag lines beneath the app logo, and essentially too much effort to say everything about the app on one page. Information overload! I suspect it resulted in people reading nothing, never finding the “sign-up/sign-in” button (the only real call to action that matters on that page, after all), and not understanding what the app does. My stats seemed to indicate this.

Slideshow only – I pared back the initial approach and used only a slideshow instead. I’ve had feedback that some of the app screenshots look great, once you’re into using the app and there is data to visualise, so I decided to show such images off and have the sign-up button as the only other noticeable component on the page, thus highlighting that call to action.

I think the problem with this approach is that it offered little opportunity to learn about the app before signing up, aside from seeing some pretty pictures. Again, my stats seemed to support this conclusion.

Rolling Demo Loop  – I thought this approach was clever. But, as we know, clever doesn’t always mean effective. I basically had the actual app itself running through a demo loop behind a semi-opaque modal panel, upon which the sign-up button sat. This way, potential users could see the real app going through the motions before signing up.

The weakness with this was that it didn’t explain quickly enough what the app does. If you didn’t happen to catch the right portions of the demo (and why would you stick around that long?), you may miss discovering that the app fits your particular niche nicely, and leave with the assumption that it doesn’t.

Expose Functionality Pre-Signup – I theorised that letting people play with the app a little before sign-up might help, and so I exposed the wizard via which you explore and create new activities and trackers (the core concepts behind Changes That Stick). Some of my existing users had commented that they liked this piece of functionality, and so it seemed worth exposing.

Once users reach the point of having almost defined a tracker, they are prompted to sign-up, after which the tracker appears in their account, thus avoiding the initially daunting blank home page too. I suspected anyone getting to that point would feel suitably intrigued to continue and sign up.

My stats suggest that this increased engagement, and plenty of people played with the wizard, but sign-ups didn’t improve much. At this point I became intrigued, started looking at this as more of an interesting experiment, and I became determined to figure out what clicks with people and pulls them in.

Quick Tour – I figured that I needed to combine exposure of interesting functionality with a more traditional overview of the app. So for anyone who doesn’t feel they have enough info about the app from playing with the pre-sign-up functionality, there is now a “Quick Tour”, which pops up a sequence of images annotated to describe the app at a high level. This may be more digestible for casual explorers.

I hope this combines some of the great images that other folks liked with a description of what the app does. And yet, by having to choose to see this tour, it won’t get in the way of anyone who just wants to dive in.

My early stats indicate that people engage with this and the wizard, but I’ve yet to see concrete proof of any uplift in sign-ups. It’s clearly early days.

Additional Imagery – As one final tweak, on desktops I noticed that the screen real estate to the left and right of the central content was wasted. And so, for users on devices with sufficient space, I added some imagery on either side of the content, pre-sign-up. Again, this is just intended to add some interest, demonstrate the visual side of the app, and increase the appeal.

I hope others find the above useful. It’s certainly an ongoing experiment and proof that, even once you reach MVP stage, there is still much tweaking and tuning to be done to determine whether an idea has a market. I believe you have to go through this before making that decision. After all, the landing page is sadly only one component of the app… and it would be nice for as many people as possible to play with the real thing before deciding whether they like it. Those are surely the numbers we’re after to really qualify the idea.

I suspect the biggest hurdle I’m facing is that my app is quite generally applicable, and therefore requires some explaining. Explaining doesn’t seem to be something that landing pages lend themselves to very well.

Building Something Great, versus Something Popular

I used to think they were one and the same thing: Building a great product, versus building something popular that people want, use and talk about. I suspect this confusion still proliferates right through large corporations, start-ups and one-person teams.

As a software engineer, my temptation is to be blinded by building a great software product. To me, “great” means solid, reliable, scalable and making use of at least some of the latest available technologies. What we often forget to notice is that the things that make a product great, and those that make it popular, can sometimes be entirely different things. Sure, there is sometimes much cross-over but they are two very different goals.

Choice of programming language, for example, probably often has zero effect on the popularity of the resulting product, particularly if it isn’t externally apparent… and yet engineers would never consider something written in a 20-year old language to be “great” if a more modern, cleaner, or efficient alternative is available.

“Great” seems to differ depending upon who is assessing it, whereas “popular” can be dispassionately backed up by usage statistics and or revenue.

I’ve known several companies who built a great product, felt that it was solid, clever and fitted their idea of a market need… but it never sold, or never really caught on. Everyone loved it, in principle, but their willingness to use or pay for it was sadly lacking. Conversely, we all know of many popular products that have rough edges, bugs, very little in the way of styling / UX gloss, and even seem stupid / impractical… and yet people use them, rave about them and sing their praises.

Clearly, those of us building products have got it all wrong. Well, perhaps we haven’t, but I suspect we get distracted by the aspects we can easily affect. In my case, that’s software quality, reliability, scalability, technology choice, etc. Those areas can easily keep a one-person team like myself occupied full-time and more. In-fact, I’m sure they currently keep the engineering departments of a few well-known companies fully occupied too!

What’s harder to spot is that we should be devoting the majority of our time to finding out what makes a popular product: One that fits a market need, does something unexpected that people will talk about and share, or simply one that is commercially viable. Any / all of those will do.

The lean startup approach of building an MVP helps here, as the focus is on building a product and validating / qualifying it as early as possible. In effect, qualifying that it is potentially popular. Over-engineering, over-polishing, optimising and refactoring (in essence, many different flavours of “great”) all get pushed out beyond the MVP stage, and rightly so.

This often leaves engineers feeling we’re building something that isn’t “great”. We probably just need reminding why this focus is important. I suspect that “great” can be engineered in, later. “Popular” is harder to nail, and probably needs to be the only focus until it some degree of market response is achieved, and a primary focus well beyond that.

We also need to be wary of something being popular in principle (“I like or appreciate your idea”) versus popular in practice (“I’ll buy or use your idea right now”).

Obvious? Well, perhaps when spelled out, yes. But I’m still surprised how prevalent this confusion is, and how common a cause of failure it has become.

Finally, I don’t doubt that “great” can be a worthy cause in itself. It’s just often not what people (and particularly companies) are actually aiming for.

10 Things I Learned by Going It Alone

Going it aloneI took a sabbatical year, starting early in 2012, ended up writing my own fitness-related web app and have since spent 3 months doing that again this year. Sometimes I underestimate how much I’ve learned by “going it alone” during that time.

Commercial Success Isn’t Everything – Though my projects certainly haven’t been a commercial success by most standards (yet), I found that I underestimate how much better I get each time I go round that loop, build something on my own, get feedback and grow. The 2nd version of Changes That Stick wasn’t just twice as good as the first, it was probably a hundred times better than the first. So many aspects just clicked, fell into place, looked better, worked better, etc.

Early efforts make subsequent efforts better by an order of magnitude, and you’ve got to churn out those early projects, crank up the numbers and get on that path. You often don’t see your own improvement, but others will and your work will always be improving if you keep producing it.

Commercial success comes through that continual improvement, not by somehow “getting it right” first time around.

Your Toughest Boss Is Yourself – If you think that working for some appalling middle manager is hard, try working for yourself once you’re fired up and engaged in your own project. Directing and driving your own work can be the best, and the worst, thing about going it alone. You need to recognise that you are a finite resource, despite the fact that your enthusiasm probably isn’t finite, and that you can only accomplish so much in a day, again despite the fact that your task list is probably endless. It’s easy to be overly hard on yourself, whilst forgetting that’s precisely the kind of behaviour we would push back on if a boss did it to us.

Work Hard… No, Harder Than That – At the risk of contradicting the previous point… When I first started on my own project, I found it hard to focus for considerable periods of time. I seriously wondered whether I had some form of attention deficit disorder. Then I realised, I had underestimated how much I required the external influence of employment, a boss and the expectation of results to make me work. It took me a month working on this, and using a great technique for building attention and focus called the Pomodoro Technique. Then suddenly I cracked it, and haven’t looked back. Now, at the peak of my activity on a project, I think nothing of getting up at 6am, and working 12-15 hour days with a few breaks for exercise, lunch, etc. And… more importantly, enjoying doing that.

Find Balance – Working crazy long days is fine for a while, but it’s easy to miss the signs that you’re burning out. I found it hard to step away from the laptop at the end of a day, acknowledge that I had done really well, and then just have an evening where I didn’t return to work. Leaving work, when you work at home, is something you have to make happen. The other problem is that returning to work only requires a double-click to fire up the IDE, open my backlog spreadsheet, or access my web app and start to make a list. And if you do that, before you know it, it’s midnight and you spent the evening working… again. Many of my early weeks ended up like this.

I found it most useful to power off the laptop, cook a meal, or get out of the flat entirely. Even a few hours off work meant I was fired up, full of energy and ready for the next day. Burning out sounds like something that happens to other people but, when you work alone, it can easily turn into something that already happened to you, at which point it’s way too late.

MVP is All That Matters – The fastest way to find out if your idea has wings is to focus ruthlessly on building a Minimal Viable Product (MVP). I did this by keeping a task list, in spreadsheet form, with a rough estimate per task, and by marking each task with whether it formed part of the MVP or not. The benefit of this is that you can get all of the tasks out of your head and into the plan, then ruthlessly change what forms your MVP as often as you like.

I found that having the spreadsheet add up remaining days of work until the MVP would be finished really focussed me. If you can see that 5 of your tasks take until the end of the month, but adding another 5 takes you a month beyond that, you’re more likely to ruthlessly decide what really needs to be in the MVP.

Prior to getting an MVP out there and qualifying your idea, any time you spend optimising, polishing, working on non-essential features (or a pet feature that you like for the sake of it) may be wasted time, as the product may not go anywhere. Why optimise for more than 100 users if you don’t find 10? Figure out the best way to optimise when you see a growing user base. Add the paid plan when you see someone wants it, etc. All of these tasks can be on the backlog list, but they probably aren’t in the MVP.

I found I needed to battle with myself daily on what constitutes an MVP. A good spreadsheet makes that possible, and far easier.

Show Your Idea Off, Daily If Possible – When you’ve worked on something alone, there’s a tendency to avoid showing your idea off until you’ve checked that it’s “good enough”. In small teams, you feel you’ve already had some feedback by virtue of working alongside others, but when working alone you fear that you’ve somehow got it wrong and yet by checking, checking and checking again you postpone the moment when you have to show it to someone and risk that they’ll say “What’s this?!”. I’ve learned just to get it out there. Email, Tweet, share the link and ask people what they think of it. The worst that can happen is that they’ll hate it, or not reply… but even then, you learned something.

I often took a laptop demo and showed it to someone over lunch, or an iPhone demo and, best still, handed the phone to them. Get it out of your hands and into theirs! Scary, but that’s precisely what will happen with real users.

It’s worth remembering that feedback isn’t always helpful, but it is all worth noting down. It’s often more useful in hindsight and in aggregate than as and when it comes in. Other techie friends often gave me a list of 1,001 things to do with the implementation, with a complete lack of feedback on whether the product met any real world need. And vice versa, non-techie friends often didn’t understand some fairly basic concepts, but that was often useful in determining the language to use, how many features to expose, and who to pitch the product at.

Check The Map – It’s easy to get lost in the detail, when you work alone. No-one else, from the same or a different group, comes along with a question, a request to look at a different angle, or just something that makes you change focus. Working alone, you need to develop the ability to switch from micro to macro focus, daily if not hourly. The feature you’re working on has a place in the overall project, which in-turn has a place in your overall plan. Regularly checking that bigger picture means there’s less chance of you ending up miles off course, adding something that really shouldn’t be in your MVP, or missing a crucial feature.

The same goes for regularly showing off your work and getting feedback; it serves as a way of forcing a different level of focus. It’s amazing what becomes apparent when you step back, see your work from the point of view of a different role, or hear what a potential user thinks.

Wear All the Hats, One At A Time – As a software engineer, it’s easy to assume that your only job, working on your own project, is to have the idea and crank out the code, but there are many other hats that need to be worn. Ignoring the fact that you will also be the marketer, the salesperson, the admin, the designer (those nice buttons don’t design themselves), the tester/QA, the documentation writer, the social media contact, etc… means that those jobs just don’t get done. Most projects don’t market themselves, there will always be a certain amount of admin, you need to do some thorough testing before each release, and you need to think of and Tweet a bunch of interesting stuff daily to invite folks to come and see what you built.

I found it most effective to acknowledge how many hats I needed to wear, and to make time to actually switch between them. Allocate a few hours to marketing, or 30 mins each morning to pre-writing your Tweets for the day (Buffer is great for this). If you know the other hats are getting some of your time, and being dealt with, it’s easier to focus on building a great product.

Users Will Do Unexpected Things, So Track Usage – Once launched, the only way to tell if people are using your product, and how they’re using it, is to embed some form of analytics and tracking. In my case this was Google Analytics, but I also got the system to email me when certain things occurred: When someone signed up, an error occurred, and a daily summary of sign-ups / return visits / dormant users.

This tracking allowed me to figure out where users got stuck and what prevented them from signing up or returning. I ended up moving a portion of the post-signup functionality to be before sign-up, so that people could play with it and see what the product does. This improved signups slightly, but it also gave users something to talk about and to reply to me about on social media.

Going forward, analytics are the only way I’ll know how/if people use the product, and where they get stuck. They rarely email you with “I would have signed up, but I found XYZ difficult”, so you need to design your tracking to tell you that automatically.

Be Persistent and Keep On Starting – Some days are good, others are downright difficult, and most are somewhere in between. One thing is constant though: you need to act as your own motivation. Whether mid-way through a difficult feature, 3 days post launch and wondering where my users are, or thinking up ways of promoting the project on Twitter… I came to realise that persistence was the most important factor in getting the project finished, and in achieving a great result.

Some days, you see a mountain of work ahead of you and the danger is you’ll become despondent and procrastinate the day away. I just got good at starting; sitting down and starting to do the next task. And before I know it, I’m 2 hours into it, and then it’s done. Starting, every day, is a skill worth mastering.

The Imposter at the Keyboard

handI’ve often talked, and joked, with a former colleague and friend of mine about Imposter Syndrome: the belief that you are a fraud just waiting to be found out, and that your prior accomplishments are the result of luck and timing rather than genuine ability or skill. Though rarely admitted, it seems to be rife in the software industry, particularly amongst those who are actually outwardly very successful. In-fact, I’m convinced the industry probably unintentionally causes this genuine psychological phenomenon.

Firstly, let me say that I am not a trained psychologist. I just find it interesting that this phenomenon seems to crop up regularly in discussions with industry colleagues and seems to be suggested, by way of implication, even when it isn’t openly acknowledged as a factor.

The sheer range of skills that software engineers could employ in our working lives leads to the obvious implication that no single engineer can be competent in all possible skills. There are simply too many to even be aware of them all, let alone regularly use them, let alone to be competent in them. Therefore, it is just a hard fact that most capable engineers will lack a significant number of skills that are considered desirable or current at present, and that this is not a reflection of their worth as engineers. As an engineer, if you look at the set of all possible software languages / skills / disciplines, you will currently be lacking a large number of somewhat recent, marketable, relevant and possibly desirable skills. Even if you actively learned 10 of the most bleeding edge ones this month, it is fairly likely that you would still be lacking some new and desirable skill 6-12 months down the line – probably the ones required by the new start-up you’ve just heard about and would love to work for. Such is the continual progression of our industry. This is also, however, what draws many of us to software engineering as a career.

And so we specialise, or we pick an area, through necessity and practicality. But what’s interesting is that, because of the grey lines between areas, we are forever discussing which skills we have, which we lack, and accounting for our level of competence in them. This mental accountancy exercise, and the introspection it brings, is ongoing. We know that the skills we picked today will become dated tomorrow and that each skill has a shelf life.

The answer to the skill competency question is never clear-cut. One person’s “senior python developer” is another person’s “intermediate python developer”. Even a senior person may have focussed on a subset of the overall skill, such that they have a blind spot in the precise area that a project requires. Java developers often suddenly discover they lack sufficient knowledge of threading and concurrency and, most capable developers use this discovery as an opportunity to learn, and to fill in that knowledge gap. There is an endless “Am I good enough at skill X?” going on. At least, in the heads of most genuinely capable engineers. I’m sure there are others who never give this a thought, which is also why I believe that Imposter Syndrome may be more prevalent amongst genuinely capable folks; the ones who think “How am I doing? What could I improve?” on a regular basis.

I suspect that bravado also gets in the way of healthy introspection. Many engineers are reticent to acknowledge the lack of a particular skill, seeing it instead as a form of weakness. And in a world where only the best get hired, who wants to admit what they don’t know? So discussions about skills are somewhat externally skewed, leaving an individual wondering whether they are, in comparison, somewhat lacking.

The path to becoming competent in a new skill is often difficult, and there is the usual Catch-22 of being unable to get on to a project requiring Skill X until you have demonstrated Skill X… on a previous project. As hiring managers, we tend to “body shop” engineers based upon skills and keywords, rather than on potential. And so we spend our personal time acquiring the required skill, demonstrating it on a side project, and somehow trying to segue from newbie to competent. There is a natural push to appear competent, perhaps before we genuinely are. Couple this with the fact that an employer will always be able to find a pool of available candidates who already have Skill X, and the urge to upsell our own skills is almost justifiable. And so it is natural to wonder whether our own hype meets the reality of our abilities.

The final contributing factor is feedback… or rather the lack of timely, honest feedback. I have rarely seen a struggling engineer told, in a timely manner, precisely the areas in which they were seen to be lacking such that they had an opportunity to improve and turn the situation round. The consequence of this is that we all wonder what folks think of us. Really think of us and our skills. It’s rarely said aloud. Again, a cause of introspection but this time not a healthy one. Perhaps if we encouraged a culture in which those difficult conversations actually took place, rather than being endlessly postponed or referred to behind the subject’s back, engineers would believe they had a firmer grasp on whether their skills were considered appropriate for a particular job. I’m lucky, I don’t seem to have been that guy… yet, to my knowledge… but I’ve known several who have heard way after the fact how they unfortunately missed the mark.

In conclusion: I think that if we acknowledged the need to specialise, to pick up new skills, to be a newbie in those skills for a while, and also learned to give honest (and sometimes difficult) feedback to one another, we might find the causes of Imposter Syndrome in our industry were greatly diminished. Until then, I suspect it will be, rather perversely, the more capable engineers who suffer from it the most.

Web Apps That Work, Even When Your Connection Doesn’t

syncI don’t know about your mobile / cellphone connection, but I know that it doesn’t take much for mine to disappear even in central London. I step inside the doorway of a shop — literally 2 inches inside the doorway — and a 5-bar signal becomes “emergency calls only”, then becomes nothing. Actually, it’s not just mobile: It happens with broadband at home and at work too. One minute a blistering-fast connection, the next my browser is timing out, images are intermittently loading, and I begin to lose hope.

So what do you do if the web app you’re developing needs to be quick, always available and not reliant upon connection speed or the factors that seem to affect it from one moment to the next?

This is the problem I had to solve for Changes That Stick, where I wanted folks to be able to quickly log activities they are tracking. It is absolutely no use if you can’t log the next step in your gym workout, the coffee you just drank, your current water intake, etc. These are meant to be quick 5-10 second steps. If they become cumbersome, slow and irritating, people just won’t use the app.

In-fact, I wanted to go further… I wanted the app to continue to work well on the London Underground, in remote areas… just about anywhere.

Your Connection Is No Good – That much is clear. You simply can’t rely on a decent connection being available at the point the user wants to perform an action in your app. It sounds perverse, but the very connectivity upon which the internet is founded is the one thing we can’t solidly rely upon as app developers.

Preload Whatever You Need -This seems to be the key. The main thing to be preloaded is the app itself, obviously, for it must live in a browser tab already if the user is going to use it when their connection is down/slow. Encouraging users to access the app and then leave it running in a tab is crucial here.

A complementary approach is to have the main app page (the one returned when they hit the server) be incredibly small, non-cached, have that page perform preloading of all other assets (CSS, images, JS) via URLs that incorporate the app version number and use HTTP caching headers to tell the device that those assets will not change. This means that, with a slow connection, even a reload of the app can be fairly swift. When the app is upgraded, the version number will change and devices will be forced to reload those assets but, within a version of the app, aggressive caching of assets is a must for slow connections.

Most of the content, logic and functionality that the user relies upon must also be preloaded into the browser when the app is first requested. This means content they see initially, and anything they may need to see in a responsive manner on-demand thereafter. This may involve periodically adding new, fresh data, if connectivity allows.

Single-Page AJAX UI – The above implies that traditional multi-page web apps, of the kind where each page is loaded as it is requested, just won’t work. You can have the appearance of multiple pages, but they must be “loaded” using JavaScript to modify the existing page DOM, turning it from one page into another without network activity. This is what we’ve come to know as an AJAX app, but it is simply HTML DOM manipulation via JavaScript.

There are many UI frameworks that claim to assist in building such one-page apps; AngularJS and KnockoutJS are great examples. But, somewhat perversely this time, I chose to write my own because my needs were very specific and perhaps quite limited: I wanted to be able to construct pages using templates, and to construct page content hierarchically by embedding page fragments within other fragments, to build up complex content and allow re-use of components from single buttons right up to entire panels. I wanted it to be possible to render a “page” using a template, then change the underlying data, and have the page the user is viewing tweaked automatically to reflect the new appearance, whether major or minor DOM changes are required to do that. Crucially, I didn’t want to have to think about making those changes, other than defining the template and the data/model upon which they rely.

So much of traditional web app programming involves the complexities of changing one page state into another. I suspect that using templates, and having the framework make the changes, removes much of that wiring. In my view, developers shouldn’t have to think about it, and the template defines the set of all possible view states.

(I’ll cover my own UI framework (“Kaleidoscope”) in another blog post soon. Suffice to say, it was a cornerstone in my quest for a one-page app).

Create rather than Consume – Preloading of data works fine for content that is known, but it doesn’t work when the possible set of data the user may rely upon is too large or cumbersome to preload. Therefore, this works best when the user will create data, or consume a small subset of data that is available at load time. Alternatively, new data may be added to the cache periodically, so that it is available when the connection is down.

Local Storage – So where do you store all that data? If you are only concerned with the data being available for the lifespan of the page, having it die when the user closes the tab or navigates elsewhere, storing it in memory using JavaScript data structures is acceptable. But as the data probably took a short while to preload, not to mention mobile bandwidth costs, we can use the alternative of browser “Local Storage”. This provides up to 5Mb of key-value storage, specific to the site that creates it (i.e. your JavaScript can’t see the local storage of another site). The benefit of using this is that, next time the user reloads your site / app, you probably don’t have to preload all of that data again (see next section).

There are alternatives to local storage, such as Session Storage, IndexedDB, and Web SQL. I avoided the latter two for now, as there isn’t widespread support across all browsers. That support is coming, and they are probably the way to go in future as they aren’t affected by the 5Mb limit. I may choose to migrate Changes That Stick data to one or other, invisibly to the user, if Local Storage quotas become a problem. Session Storage is cleared at the end of each session but, again, isn’t affected by the 5Mb limit so may benefit some specific app uses, particularly where more storage is required within, but not across, sessions.

Data Sync – So in theory, all of the above means we can write a 1-page app with the appearance of multiple pages, store and retrieve data locally, and allow the user to work with no connection. The only problem here is what happens if they connect to the app in another browser, on another device. Surely they need to see the data they’ve been editing on the first device? This is where data synchronisation comes in.

When a connection is available, data must be synchronised with the server, in both directions and preferably without the user needing to initiate it. I chose to do this via the usual XMLHttpRequest (XHR) object to both poll for new data, and send pending changes. Using a sequence number, it is possible to tell the server the last set of updates we received, such that we only receive anything subsequent. Each client needs to mark the changes it made with a timestamp, because updates to the same piece of data will overwrite one another.

Which brings me to granularity: There is the possibility that the user is editing slightly stale data. This means that their changes either need cleverly combining on the server (perhaps by sending clever “deltas” of their changes, rather than the entire object), or keys need to be used at a level of granularity at which we’re happy with overwrites occurring.

For Changes That Stick, I took the latter approach for simplicity. Individual activities have their own storage key. If you edit the same activity on multiple devices very rapidly one after the another, one edit may overwrite another. But, crucially, this means you can add two new activities, one on each device, with no fear of them overwriting one another. I wanted that basic use case where users are just adding activities to work across devices, with both devices eventually seeing both activities, and no overwrites. I also used timestamps as the unique keys for activities, making it very unlikely that keys themselves will clash.

Finally, having made data synchronisation seamless to the user, keeping them aware of whether their data is synchronised and up-to-date is important. If they try to close the browser tab in which the app is running when data is still waiting to be sent, it is worth asking them if they meant to do that. Otherwise, perhaps they won’t return to this device for a few days, if ever, and will wonder where their update went.

Other than catching their point of exit, I chose to show the “sync status” as a red-yellow-green status for the menu button. The menu also shows the last sync time, and allows the user to force a sync now (they may know they just left an area of no signal and wish to force sync before they go offline again).

Go Native? – It is certainly true that some of the above issues are easier to solve if you build a native app for each mobile platform. My own resource constraints (being a team of one) meant this wasn’t an easy option for me. I also wanted the same app to work on desktops and laptops. One possibility is to wrap my web app in a native wrapper but, again, I haven’t yet had the time to investigate this yet. I may return to this and post an update.

For now, Changes That Stick works well with little/no connection, even on the London Underground… though I’ve yet to find a use for it on the Northern Line!

“What Next?” May Be The $6M Question

For at least a little while longer, I am something of a one-man dev / product / app team and I regularly have to answer the question “What Next?” – The danger, as a software engineer, is picking technical tasks that are do-able rather than other stuff that might be important or perhaps tough, unknown or scary.

Rather unexpectedly, this is an interesting question for me right now. It comes up on an hourly basis and in discussions with friends / former colleagues who suggest possible directions. I think I’m getting better at answering it; for myself, and for them (for everyone has a strong opinion on the answer!)

Having just published a new web app, I’m faced with a list of tasks to complete. Some obvious bugs, other things to investigate more deeply, the odd performance issue, features large and small, etc. But here’s the most surprising thing: The things I should be tackling first probably aren’t even on that list.

Having basically launched my Minimal Viable Product (MVP), polishing the somewhat rough technical aspects feels tempting to me, as a software engineer. However, the next step probably involves figuring out whether the product is something people want, turning it into something they want, and determining whether there’s an opportunity to make something commercial from it. I need to work on qualifying the viability of the MVP, not just the next technical task that pops into my head and feels do-able.

One possible outcome is that the product isn’t viable, and it will just remain a portfolio project for me… but figuring that out is going to take some careful exploration, not just a blind following of tasks from my backlog. It would be a lost opportunity to polish it now, when it could have been so much more.

I’ve spent the past 2 months deeply focussed on development and technical issues and, since launch, I’ve already done things like reduce the app load time… but I’m resisting going further just yet, not because I think it’s the wrong thing, but because I think it’s a less important thing now. I would clearly only be picking it because it’s do-able. To speak in clichés, it’s a low-hanging fruit.

So some folks may walk away because the page loads in 2½ seconds not 1½. I know, and I’ll get to it. I’m more interested in the (probably larger) set of folks who may walk away because the app isn’t meaningful to them, isn’t interesting, or is a tantalisingly small number of tweaks away from being something they want… but not quite.

My resources are finite (are they ever anything other than finite?!), and there’s a danger of getting sucked into an endless cycle of “just one more technical thing” because it makes me feel busy, and the “next thing” always feels crucial, rather than focussing on the bigger issues first. There will come a time for more technical stuff… probably very soon, if I can qualify the product as viable.

I’ve got to pick my battles. Resisting my tech-first urges is certainly an interesting lesson.

Beyond the “Quantified Self”

ChangesThatStick.comMany of us, intentionally or otherwise, have played a part in the Quantified Self movement: Using data acquisition and tracking to gain an understanding of various aspects of our lives — exercise, diet, sleep — often using clever wearable devices or mobile apps.

The attraction is understandable, and who doesn’t like to navel gaze?! Finding patterns in that data, root causes or commonalities is certainly interesting and it automatically inspires a degree of change and self-improvement.

But beyond pure tracking and analysis, I think it’s crucial to commit to very specific and concrete goals, regularly compare our progress against those goals and to make genuine changes to the tracked aspects of our lives. I believe that’s where the true benefits of the Quantified Self lie: building on it with goals, willpower and tracking. It’s what I’ve come to refer to as the “Inspired Self“, and it is only possible because of the Quantified Self movement upon which it is built.

Write it Down – We are often encouraged to write goals down, because it causes a degree of commitment to them and increases the likelihood that we’ll follow up with action. Likewise, we should record our commitment to change, define up-front the criteria by which we will judge that change, and ensure that we have visibility of our progress… good or bad. These extra measures make us accountable, aware… and inspired to follow through with action. It’s one thing to quantify and explore aspects of our lives, but another thing entirely to be inspired to improve them.

ChangesThatStick.comMaking Change Measurable – Commitments to change must be measurable, to remove any element of judgement or cheating. “Five miles” means 5 miles, not 4¾ or 5¼. In comparison, “Go for a run” means whatever we feel like on the day and gives us carte blanche to modify the goal.

It’s Not Science – Goals, and the activities that contribute towards them, should be measurable in our favourite units: A commitment to cut down our caffeine intake might begin with reducing by one “cup”our favourite cup, not some standard measure. Tracking of activities that contribute to a goal should always be quick, simple and tailored to our own chosen level of detail. It’s not a science experiment, nor should it become a chore, but it must be measurable.

Path of Continual Improvement – The reason most people pay a personal trainer is that the trainer will set goals and dispassionately reflect progress against those goals but, crucially, they will also set new goals as time goes by. Committing to run 5 miles a week is fine for now, but we should probably extend that commitment to 5½ miles and beyond, as we achieve each successive goal. Achieving the original goal is only part of the picture, and setting a path for continual improvement is the longer-term aim. From where we are now, to where we want to be, in manageable increments. Having someone, or an app, automatically change our targets is even better.

There is a slow and steady (but probably achievable) path to us running a marathon in 12 months time, losing some weight in time for summer, increasing our bench press, managing 10 chin-ups, cutting down to 1 cigarette a day, etc. Today — or this week or this month — our achievement only needs to be a portion of that bigger goal, safe in the knowledge that we are on the path to the overall goal if we stick to the smaller goals en-route.

Visualising Progress – The Quantified Self movement introduced us to effortless ways of capturing data about our lives and our performance. Visualising our progress versus our goals must be similarly effortless. We need to remain aware of how far we have to go to meet them. It’s no use having to manually update a graph on our fridge door or a spreadsheet on our laptop. Our progress must be there, in our faces, whenever and wherever we want to glance at it.

Unifying our view of that progress across many goals or activities also increases the visibility. We would rather see our progress in one place five times per day, than in 5 apps once a day.

Making Better Choices – Continual visibility of our progress has a startling effect on our rate of change: Because our progress and, specifically, how far we have to go to meet the current goal, is in the forefront of our minds… we make better choices in-the-moment. We go for that extra run to meet the weekly target, skip having the cupcake (!) so that our average weight might be on-track again this month, or have a soft drink instead of another beer in order to meet our “3 drinks” weekly limit. There’s no ability to deny the goals, no “out-of-sight, out-of-mind” occurs. The long-term achievement stays in our heads, giving it less chance of being overwhelmed by the instant gratification that might lie before us. (As we know, those cupcakes actually speak to us ;-))

Handling Setbacks – Setbacks occur, targets get missed, plans change and real life sometimes gets in the way. Our visualisation of progress may show the reality: that a particular change is going to be hard. This is the downside of tracking our lives, but admitting our shortcomings is the first step to finding a way to reach the goal.

We have two alternatives when setbacks happen: Give up, or Reframe and Re-Commit.

Reframing might mean seeing that we are off-target and setting a lesser goal for the next time period. Perhaps we realise now that we won’t lose 10lbs by this summer, but we might manageably lose 5lbs, a worthy goal in itself. So we can change the goals for the future and get back on-track. Often, a certain amount of learning, reframing and tweaking is required before the path to an achievable goal is found. The initial enthusiasm sometimes leads us to set unrealistic goals (New Year’s Resolutions?!), and it’s easy to forget, when we fail to meet them, that a slightly more conservative goal may be entirely possible. The original goal may also be achievable beyond that, if we take it in smaller steps.

20130713 - starsCelebrating Success – Just as our progress versus a goal should be visible to us, so should our achievements: We want to see how long we have been on-target and the milestones we have passed. Another week on-target needn’t be “just another week on-target”: It can be “My 5th week on-target!”, which is halfway on the path to 10 weeks on-target. An entirely more inspiring way of seeing it. Independent assessment of our achievements feels like being rewarded for our hard work. We met our goals for a period of time, and it has been recognised. Something as simple as an email, or a row of stars, makes all the difference. The technical term here is “Gamification“, though I usually just think of it as “Making things fun” 🙂

As Ever… There’s An App For That – is my first venture into the world of the Inspired Self. After years of using several different tracking apps (for running, diet, weight, water consumption, exercise, habits), I decided there must be a better way to track my progress in all of these areas, in a single unified app. Often, the apps that are great at data capture really suck at goal setting and visualisation, and vice versa. There just didn’t seem to be a unified app that did the lot. The problem wasn’t my commitment, but rather the spreading of my commitment between so many different apps.

So, I wrote Changes That Stick as a mobile/web app, such that most measurable activities and habits can be recorded, goals set against them, and progress visualised nicely on one screen, even whilst I was out and about. I currently track and have goals for my weight (overall, fat%, lean, BMI), water consumption, running (distance, time and speed), gym workout (each exercise, with weight, reps, sets and notes), blood pressure and the odd habit or two that I’m changing. But it can also track so much more, and my aim is that there should eventually be a way of tracking and setting goals for whatever need people come to the app with.

I wanted it to be flexible enough to track my activities in appropriate measurement units. So, finding gym equipment marked in pounds, when my goal was in kilograms, wouldn’t see me dashing for my iPhone calculator to manually perform a conversion. The app knows about most measurement units, and converts between them. I aim to add units I’m missing, and you can describe any unit used simply for counting up activities using any phrases that you like, so if I want to track my weekly run in nautical miles, or count a path to quitting smoking in cigars rather than cigarettes, that should be possible.

I also wanted to track derived aspects of activities, such as my lean weight, though my scales only captured my overall weight and fat percentage… another easy problem to solve with software, and so I did. This means I can set targets to increase my lean weight, whilst I decrease my fat weight, all by capturing my overall body weight and fat%. I can track my pace per mile from my running distance and time, or my maximum reps this week from one of my workout activities. The data I capture, and the data I want to track the goal against, are often subtly different, but it’s possible to derive one simply from the other with some simple knowledge about how they are related; knowledge that went into the app. Again, something so often missed by other apps and, just like ability to use our favourite measurement units, it makes the goal so much more meaningful to us.

The app allows targets to be changed over time, so a path of improvement is defined. I would like to increase my lean weight, but decrease my fat weight… and I have gradual changes in target to stick to each week, which will take me to those longer-term goals. I’ve done similar with other things that I’m tracking.

I hope to integrate the app further with other Quantified Self apps and data sources, such that activities people are already tracking through their favourite apps may be automatically captured for them, used to set goals and visualised. People like to track their running on an app that has GPS input and displays a map, enter their calorie consumption on an app with a great database of known foods, or track their movement using a device that clips to their belt, and so they should continue to use those methods… but they might like to set goals and visualise their progress against them in one unified place. Something of a “how am I doing?” dashboard, to help improve those in-the-moment decisions.

On a technical note, and as a one-man development team: Rather than producing different versions of the app for iOS, Android and desktops, the same app runs in any HTML5-compatible browser on most desktops, mobiles, smartphones and tablets. I hope this means there is some consistency across platforms and I can incorporate new features rapidly across them all. Basic use will be free, and there will be a paid plan for more extensive use beyond that, with a low monthly subscription cost to cover my hosting costs.

I hope that being Quantified, and Inspired to change, will be a powerful combination. Manufacturers of cupcakes will no-doubt hate me 🙂