Anatomy of a “No Frills” Web Application

This isn’t one of those “best practice” or “how to”  blog posts; the ones where you suspect the author has architected but not actually built a real app for a while. Instead, it’s rather an unglamorous overview of the architecture of UbiquiList.

A few folks asked me which tech choices I made, and a few non-tech friends showed an interest in seeing what goes into an app like this, so I hope this goes some way towards covering both angles.

I suspect that you could build the same web application using 101 different technologies that I haven’t even mentioned here, and combine those technologies in 1,001 unique ways. Some of these would no-doubt be considered much “cooler” than the architecture I describe here. And yet I bet the end result would be uncannily similar from a user’s viewpoint. This is the reason I try to stay out of those almost religious debates about web technology choices. Beyond reading up or comparing notes with others on what’s new or improved, and understanding which choices impact performance, hosting costs or user experience in a tangible manner, I like to spend the majority of my time building things that people actually get to use. Some of my choices here may not match yours. As always, I learned great deal and I will do many things differently next time.

I am staying, out of necessity, at quite a high level here too. I could write an entire post on each of the topics of caching, persistence, performance, load balancing, sticky sessions, AJAX, etc. But this is more of a map than a detailed account. Maybe I’ll cover some areas in detail in another post.

As applications and products go, Ubiquilist is somewhat at the MVP stage, and is therefore lean and very minimal in nature. Some of the architectural decisions I made were for expedience, and relied upon prior work that I could re-use quickly. There is also considerably more “architecture” here than most MVPs would employ, for the same reason: I had it easily to-hand. Most MVPs would skip a lot of this and revisit the need for it later.

Most things are easier to explain if you start with a picture. So here’s a picture…

Architecture - blog post

Single Page rather than Click-Thru – The first thing to point out is that this is an AJAX / single-page web app, rather than what I tend to call a “click-thru” web app. This means that the app dynamically modifies the page content to reflect its internal state, rather than requiring users to click links that cause the server to return an entire new page for each action. Single page apps seem to be the way things are heading.

Kaleidoscope – To build such a single-page app, I used a framework of my own called “Kaleidoscope”. There are several great alternatives out there, such as Knockout, or AngularJS, but I wrote Kaleidoscope (“KS” from now on) for a previous project and so it was the simplest choice for me. I’ve been judged as crazy and brave, in the same breath, for building a framework. It served an educational purpose for me, and the result is something I can build apps with fairly quickly. I probably wouldn’t recommend writing your own framework though. It’s a huge distraction, and there are plenty of existing ones out there!

In a nutshell, KS allows me to quickly build interfaces constructed from a hierarchy of XHTML “fragments” by writing a simple “builder / controller”  for each fragment. When requested, a builder provides a JSON model representing some portion of the app state. It also provides an XHTML template for the fragment, which references the model and allows the fragment view to be built by KS. Whenever the model is updated, the builder simply supplies the updated model and the framework handles any changes to the page DOM dictated by the template, whether small tweaks or huge structural changes. There is no need to be concerned with the mapping of one page state to another or updating of the DOM, and KS removes that significant burden from the developer… hence my love of using it, and the reason that fairly dynamic-looking apps can be built so quickly with it.

Templates and Actions – KS templates may contain conditional and iterative logic via JavaScript “scriptlets”, much like JSP, meaning that the fragment view may change either subtly or radically based upon any aspect of the model. (A few folks hate the idea of mixing content with logic… but I say it’s practical and they always seem to fail to come up with suitable alternatives).

KS templates may also embed other fragments, allowing the framework to orchestrate those fragments into an overall page view; something else the developer doesn’t need to be concerned with.

Finally, Action tags in templates allow chosen HTML DOM events to be passed back to the builder as application actions, which the builder handles in an app-specific manner, perhaps further updating the model and hence, indirectly, the page. For example, clicking on a button or changing the selection in a drop-down, but also subtler events that may not necessarily change the state of the app directly, such as mouse movements or keyboard events.

Modules – The client-side app is split into modules — about twenty of them — each with their own API. This is fairly standard when JavaScript projects get beyond a single source file and it helps to separate concerns. Aside from the application and UI logic itself, there are a couple of utility modules and a module for all client calls to RESTful services via which the browser-side app calls the server (see below).

The entire app is further combined during the build process into a single JavaScript module, with a simple API for initialising the app exposed to the page via an “LI” namespace (“LI” because I hadn’t decided on the app name yet, and I knew it would be concerned with lists!). The resulting single, rather large, JavaScript file is then passed through Google’s Closure Compiler during the build, both to obfuscate the app from prying eyes and to significantly reduce its size. The end result is a single JS file containing the entire client-side app.

Server – All of the app’s server-side components are hosted on a Amazon Web Services (AWS). They now offer such a bewildering variety of components, great scaling options and a year-long “free tier” to get you started cheaply. There are other providers, but I found it easy to get started with Amazon and I know the app can grow without moving from them. Again, another piece I had used before, so I stuck with them.

Hosting Static Content – All images, CSS and the client-side app JS itself are stored on Amazon’s S3 storage service, versioned by build number. When uploading static content to S3 as part of a build, I also set file type and caching metadata using a simple script. In-front of S3 sits CloudFront, which acts as a Content Delivery Network (CDN) and optimises access to that static content via an edge delivery network, depending where an accessing user is located. The app automatically precedes HTML and dynamic references to static resources with the path to CloudFront and my aim was to avoid serving any static content inefficiently from EC2 (see below).

EC2 – All dynamic portions of the app are hosted on one or more EC2 instances, the number of which can be scaled up to meet demand. Those instances sit behind an Elastic Load Balancer (ELB) which automatically forwards requests to “healthy” instances. The ELB is also where the site certificate is installed, allowing secure HTTPS communication between browser and server. In-fact, after the first page hit, UbiquiList mandates a secure connection from then on.

Each EC2 instance is a Linux box running Jetty, upon which the Java server-side application runs. There are other servlet containers, and full-blown application servers available, but Jetty serves the purpose very well for this. My alternative would probably have been Tomcat.

The server-side app is mainly comprised of RESTful services, via which the client-side JavaScript app talks to the server. Most services return a portion of the server-side model (lists, list data, or user data) in JSON form, and allow that model to be acted upon by the client via further actions (create list, modify list item, etc). Another service deals with security, allowing users to sign up, log in and log out, and this administers the associated cookies that are required in order to access the user’s data securely via the other services.

There are also a few bootstrap services, via which the app is launched, or that just don’t fit into the RESTful approach. These accomplish tasks such as handling the first page hit containing the initial skeletal HTML, referencing the correct JS libraries and any environment-specific configuration. They also provide a destination for folks that Kaleidoscope deems to have an old / incompatible browser, and a suitable page is shown to such users explaining why.

Model, Caching, Persistence – The entities that make up a user’s view of UbiquiList (lists, items, cells, other users) are represented by a server-side model. When an entity is accessed via the client, a server-side representation of it is either created or loaded from the database.

To avoid repeated database access, a caching layer is present. The eventual plan is to use Amazon’s ElastiCache to host a shared cache that all EC2 instances access. But, for now, whilst there is a single EC2 instance, an in-memory cache is used instead. This was part of the whole MVP / early version approach, in which scaling decisions are postponed. I do, however, believe in figuring out where scaling might be needed and making sure it can be slotted in later.

MongoDB – Having used MySQL as a database for the past few years, I chose to move to a NoSQL approach this time and went with MongoDB. It isn’t hampered by so many of the scaling issues that traditional relational databases suffer from and, having never used it before, it seemed a great opportunity to give it a whirl. To free myself from the initial learning curve inherent in hosting my own Mongo instance, I chose to go with MongoHQ who provide hosted MongoDB installations. They have options that are hosted within Amazon’s own AWS infrastructure, hence traffic between the EC2 instances and MongoHQ is internal to AWS. So-far, I’m loving the simplicity of both MongoDB and MongoHQ!

Third Party Services – Finally, there are a few third party services upon which UbiquiList relies. MaxMind is used to map a user’s IP address to their geographic location; something that people take for granted with web applications nowadays. I use one of their web services and pay a small fee per 50,000 lookups. This allows UbiquiList to display the location of anonymous / guest users, and helps to assign a basic description to them in the event that they don’t provide a nickname.

Mandrill provides a great email gateway, via which all UbiquiList emails are sent. They support configuration of the appropriate DNS settings, such that emails from Mandrill are known to come authentically from the UbiquiList.com domain. They also monitor undeliverable mails, and provide a whole host of features I’ve yet to explore, such as tracking of email campaigns.

What Else? – I have probably missed out a thousand things, such as Google Analytics, gzip compression, precisely how I upgrade/deploy builds, etc. But I think this is more than enough for an overview.

Where Next? – The architectural decisions I make next for UbiquiList really depend on where it goes as a product. The architecture I describe above is working well as an MVP platform, but I suspect (in-fact, I know) that if the app needs to scale up, I will need to make changes in unexpected places. That’s a great reason for keeping the architecture fairly minimal for now and avoiding components that may never be needed.

What would I change? – Right now, I’m not seeing any components of this architecture that stand out as weak or poor decisions. I know that will change rapidly at some point. I still stand by Java as a server-side language, despite premature talk of its death (don’t even start me on “fashionable” programming languages here). Similarly, whilst I’m not a massive fan of JavaScript, it is (for now) a necessary component of any full-stack web developer’s toolkit and, if you ignore the inconsistency of the DOM model and just look at the language itself, is a great language.

I hope this bare-bones description of the architecture of UbiquiList has been useful and, if you’ve reached this far, thanks for wading through it 🙂

The Tech Book that Inspired Me Most

51fp13jaqFL._When I was around 10 years old, I asked my Dad to buy me a copy of “How to Build Your Own Self-Programming Robot”, by David L. Heiserman. At the time, it was way too advanced for my abilities, and everyone warned me of this. But I kept asking, and so they bought it for me. Probably just to shut me up.

Owning this book did something amazing for me.

Firstly, I never built the robot. In-fact, my Dad made it clear that wasn’t an option, as the cost of the parts was phenomenal back then. But what it allowed me to do was aspire, technically, and it also encouraged me to learn. I lapped up the contents, even though portions of it were totally beyond me. On repeat read throughs, I understood more and more of it each time. I went and looked the parts up in a Maplin catalogue (another source of fuel for my aspirations), and discovered a world in which it was possible to build… virtually anything.

This book taught me that the only way into that world of tech possibilities was to discover it myself, read beyond my current understanding, to learn, and to try things out. Basically, to stretch myself. And not to listen to the advice that I “couldn’t”.

I was reminded of this when a friend recently posted on Facebook about how he is programming a Z80 microprocessor, and I got a sense that he was really enjoying that. Suddenly, all of that early excitement, the possibilities, and the aspirations came flooding back to me.

I suspect that 32 years later, as I continue working on UbiquiList and aspiring to make something of that, the advice I often get to go back to a “normal” paying job is just an echo of the very same advice telling me that this book was way beyond me.

Luckily, I haven’t stopped aspiring or trying or stretching my knowledge and abilities. And this book reminds me not to 🙂

UbiquiList, and Why I’m Reinventing the Wheel

mundaneWe generally seem to think that the best product ideas will be found in areas that are new, novel and relatively unexplored by others. Whilst it’s true that the most newsworthy products, apps and websites often fit this pattern, I think there’s a great deal of mileage in building products to address pain points in far more mundane and familiar territory.

I just spent the past 30 days building an MVP of a tool called UbiquiList to simplify list-based collaboration, with the aim of finding places where it removes the tedium and the unnecessarily manual aspects of certain day-to-day tasks in our working lives.

Of-course, particularly in the tech industry, there are already 1,001 tools out there for sharing documents, collaborating on plans, and building lists… so this territory is not only explored, but rather well-worn.

Why am I even bothering to build something that surely competes for the same users? Because I really don’t feel that it does.

For a subset of potential users of this kind of tool, most of the existing products in this space are either overkill, considered too heavyweight and cumbersome, or fulfill way too wide a brief and are therefore either considered poor value or just plain scary. This is evidenced by the sheer number of work-related tasks that people still perform manually, where some element of the task could be greatly eased by use of a shared list: tracking meeting actions, planning work, building an agenda, following a procedure, coordinating across time zones / locations, or just brainstorming new ideas. Particularly outside tech environments, tools are rarely used to improve these use cases.

I wanted to build a tool with which someone could quickly throw together a list with a column format of their choice, then email / chat / text a unique link to anyone — irrelevant of boundaries and not requiring that person to have an app account — so that they very quickly have a group of people, perhaps even in different locations, collaborating on items in that list. Full-stop. Nothing more than that. In-fact, the more you add to that basic use case, the less likely that it fits those simple pain points so neatly. It also begins to encroach upon areas that other products have successfully tackled: project management, planning tools, etc. I feel the potential success of this idea may perhaps lie in its simplicity, not how many bells and whistles it can incorporate.

I wanted something simple and ubiquitous, that could be used regularly, at the drop of a hat, in the same way that we use Skype, or arrange meetings in calendars, or use group chat. These are probably not places for novel, new products… these are places for something that just fits and solves a problem neatly. This is definitely mundane territory for most, but I’m rather excited by the possibilities and how widely applicable it seems to be, given the market research I’ve already done.

The challenge is that I think the overly-manual tasks that it is perfect for are considered so obviously mundane that most people don’t even consider using a tool to improve them… perhaps because they see these tasks simply as unavoidable. I need to find a way to pitch it as so simple that it springs to mind whenever these pain points arise. Of-course, the great thing is that it also advertises itself to a certain degree, by virtue of encouraging sending of lists to non-users.

It’s an early MVP and, as ever, this is just the starting point for a discussion about what works, what doesn’t, and how to develop it as a product. I will be posting much more on UbiquiList over the coming weeks.

The 1 Big Reason Why Everyone Should ‘Go It Alone’ At Some Point

person - iancackettFrom childhood, we are taught to “play nicely” and to get along with others. At school and in our professional lives, we are encouraged to work as a team.

I sometimes wonder whether this incessant focus on teamwork leads to us missing out on discovery of a different, and in my view equally important, life skill: Working alone.

Very few people have truly worked alone. By that I mean picked a project, planned their work, plotted their own path towards a deadline, been their own reviewer, their own source of feedback, and been 100% responsible for the work they delivered. Sure, we’ve all contributed our own work within a team, but do we really have a handle on the value of our contribution and how to improve it?

Some would say that, so long as the team succeeds in its goal, all else is irrelevant. There is also a certain stigma attached to being good at working alone… as if it implies an inability to work with others.

I suspect that by only seeing the effect of our own work in aggregate — as part of the overall team deliverable — we never truly get a chance to identify our own strengths and weaknesses, and therefore to grow. In that sense, if we always work as a team, we never really get to fine tune the one thing we bring to that team: Ourselves.

I’ve written before about the things I learned by ‘Going It Alone’. Whilst I haven’t had a commercial success yet, I think my understanding of what I’m capable of, my self-management, and therefore my ability to make a better contribution to a team, has improved immeasurably.

I suspect that most people, at some point in their working lives, would benefit from a period — even a short one — working on a project of their own. And by that, I mean truly coming up with an idea that they hope to be successful in some way (whether commercially or purely by being useful to others), pursuing it until they have a finished deliverable, taking it to a market or audience, being 100% answerable for it and running with it to see what it can become.

I think that people would bring more to teams if they understood how to improve their own contribution, and I don’t believe you can always do that whilst being part of a team at the time.

Perhaps, in addition to writing the rather tired old phrase “works well in a team” on our CVs / resumés, we should be looking to add “also works well alone”. I think it’s an underrated skill and can have a knock-on effect on our ability to work with others.