Broken. A story of Online Development

Image generated using Dall•E 2 Artificial Intelligence from 2 phrases from Virgil Reality.

I’m not an absolutist when it comes to technology. Everything works some of the time. However, some legacy systems are held together by code that is between 10 to 25 years old, (63 years in one exceptional case). Given some of the ‘kludged’ together code, the market seems pretty OK with a significant slice of ‘broken’.

That’s not to be arrogant. The fact that anything exists and works at all, is amazing. It’s due to a lot of hard work, innovation, and the changing fortunes of a market driven development roadmap.

It’s all how you look at it. All software platforms can, and do have flaws. Legacy bugs and issues introduced over years of development. This is part of a phenomenon known as Tech Debt.

When you use particular platforms for development, you inherit that debt into your own project. OK, cool. On the flip side, that platform may save you untold time in not having to develop those features. Depending on your project, you can live with it, or plan for tech debt.

Other times the missing or broken feature, may be the undoing of a business venture.

All code languages are also abstractions from the binary (01010101) or assembly language that a computer chip uses to process instructions. Even if you build it all yourself, using a particular language or framework, your project will still have tech debt associated with it.

Many online articles are absolutist and present a world of ‘this’ vs ‘that’. They present a development universe where picking a particular platform (e.g. WordPress) or supplier, is the first step. Without the right planning, this is a poor choice.

Sometimes there are developers who like a particular product because they know how to use it. Thus, even if an article contains a lot of great information about a platform, it has an undercurrent of shilling for that platform.

There’s a simple way of looking at platforms and tech debt that may give you some strategic benefits for your next project. One that focuses on your needs and success.

Consumption

Let’s start with what you can eat. Any development project you initiate will have to be consumed. That means you’ll have to be able to integrate the outcomes of any project into your organisation.

You and your team have to be able to plan, implement, use, and provide the feedback loop. You have to make the return on investment worthwhile.

At this stage there is no reason to even discuss how. This is purely the reasons for, and the expected outcomes.

Are you ready to take on the project? Is your team ready? Do you have the cooperation, buy-in and enthusiasm of everyone you need? Even if it’s just you; have you considered the outcomes being sought?

How are you going to integrate the development project and the delivered site, app or system into a business? Have you anticipated scenarios that include an influx of customers, impact +/- on customer service, or the time just to manage the new system. Are there contingency plans?

Gartner research suggests that 75% of IT projects fail (2017), 87% of data science projects never make it into production (2019) and through 2022 only 20% of analytic insights will deliver business outcomes (2019). This has been a consistent story of IT projects over the years.

At a basic level, you’ll need to think about:

  1. Intended use: Working backwards from the intended business outcome.
  2. Technical expertise: Do you have the expertise, or do you know how to onboard and manage that expertise?
  3. Budget: Your final scope will drive your budget. Get more than one quote. Assess your ability to raise capital and funding if in a startup. Get advice.
  4. Functionality: You know the outcomes, what are all the functional parts that make that come true?
  5. Ease of use: Has everything been scoped properly to make this a great experience? What is the learning curve? What happens if something goes wrong? How easy is it to build on top of this version?

Given that the requirements are the recipe for your development. Your consumption plan. That plan should consider business ROI (Return on Investment), architecture, design, development, code management, testing, documentation, infrastructure, people, process, service and time as part of the technical debt.

This is what you expect to happen when everything goes right. Minimising tech debt is the start of making sure things don’t go wrong.

Tech Debt

Knowing how to consume a project is a process of focusing inward. Turning this focus outward allows you and your team to consider those things that cost. It allows for decisions on when and where technical debt will sit in a project.

Technical Debt as a concept was coined by developer, Ward Cunningham. He was one of 17 authors of the Agile Manifesto and has also been given credit for developing the Wiki.

Tech Debt is an implied cost of additional work caused by choosing easier (limited) solutions up front, rather than using an approach that may take longer or require better planning. It also refers to refactoring code as usage and experience with the codebase goes on.

“With borrowed money, you can do something sooner than you might otherwise, but then until you pay back that money you’ll be paying interest. I thought borrowing money was a good idea, I thought that rushing software out the door to get some experience with it was a good idea, but that of course, you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.” — Ward Cunningham

In that case, tech debt does not refer to debacles. They are simply debacles. Tech debt is a more considered and manageable concept as part of development.

If tech debt is managed badly, it can cause projects to miss deadlines, costs to blowout, or even project failure. Deadlines are missed when there are more things to do (tech debt), than there is time to complete the work. As a project matures, or has been around a while, tech debt can build up.

This may be adding new features onto an existing codebase, changing development team members, changing suppliers, fixing issues or patching exploits.

By not paying down this tech debt, the delivery of functionality slows. More people might be needed, and more bugs introduced. Team stress increases. Estimations may be less accurate. This may allow competitors to take over the niche that a product or brand holds.

Releasing code into the live environment also has challenges. You can’t take features in use offline. If changes cause other functionality to fail, or there are outages, the risk of financial losses, ill will and legal issues increases.

Product managers or business owners do tend to know about technical debt even if they don’t give it that label. The issue comes in underestimating the consequences of carrying large amounts of tech debt. The debt could be one that you can put off a long time, or may cost a lot of time, in a short time.

Platforms

By this stage you should have documentation describing your project. It specifies the Business, Functional and Technical Requirements. This can now be built upon by matching platform tools that will suit your development, to the requirements.

Picking a supplier or doing this before you have that in hand, is largely responsible for the statistics presented earlier. You could be part of the 13–20%? Maybe?

At this stage, you’ll also know roughly the kind of development platform you’ll need and this will make your comparisons easier, keeping the process in your control.

Platform Categories

I categorise technology platforms, so projects and platforms are easy to discuss. We can discuss if one or more approaches is required.

Within each category may be a number of competing products. Content Systems for example.

Working consultants or potential vendors, you’ll have a set of questions from your requirements. You can use these questions, along with your own research and knowledge, to match the platform to your project.

This categorisation, also gives a framework to discuss current platforms used within an organisation.

They are:

  • Tier One — Hosted Platforms
  • Tier Two — Wordpress
  • Tier Three — Database and Middleware
  • Tier Four — Prototype, Low Code and No Code Platforms
  • Tier Five — Static Generation and Code Frameworks
  • Tier Six — Apps
  • Tier Seven — Insight, Data and Analytics Projects
  • Tier Eight — Enterprise Systems

Tier One — Hosted Platforms

When these platforms first came into existence, they were sold on the idea that you didn’t need a developer, and could do it all yourself.

As they have added more sophisticated elements to compete with other platforms; surprise, surprise; many of them have spawned developers, who specialise in these hosted platforms.

I think the range spans from using a Facebook Page, a hosted builder (such as SquareSpace [2003], Wix [2006] or Weebly [2006]) or more sophisticated shopping platforms such as Shopify (2006) or BigCommerce (2009). There are others, though that’s the basic idea.

Most have a commerce offering tied in now or as part of the central premise. Shopify has a nice bit of kit for integrating the shop into a Facebook page.

When you use one of these platforms, you inherit how they function, how they change and the development roadmap of the platform company. That has to be factored into your risk and tech debt strategy.

Tier Two — Wordpress

WordPress (2003), sits in a niche of its own. It does use an interface connected to middleware and a database like Tier Three. It also has a hosted option. Simple enough for many integration developers to install, add a template, add plugins and get a website up and running. Depending on scope and skill, it is totally customisable.

You might not build your new product platform on it, though for a great number of cases a sophisticated developer can do a lot with it, or sit other apps on the same server. Wordpress also has a well known commerce plugin called WooCommerce.

WordPress has a lot of baggage that comes with the ‘making it easy’ or plug-in part. The codebase does make a lot of work speeding up sites, given the plug-ins, themes and page builders used. Even if a lot of it is being rewritten to replace PHP with JavaScript that only serves to introduce more potential bugs.

Tier Three — Database and Middleware

This includes most Content Management Systems (CMS). A content management system usually consists of a web interface, middleware to communicate with a database and the database itself.

Like WordPress many evolved from blogging platforms, where content was seen as the key to interest on the web. Now I think the emphasis is more on utility value, where you go to a site or platform to achieve something. Meaning you choose a CMS when content is your primary business or a valuable part of delivering the service (e.g. industry knowledge). This also includes CMS systems where there is no front end templating and the CMS is considered ‘headless’.

Some example products may be Drupal (2001), Joomla (2005), Expression Engine (2005), Craft (2011), Umbraco (2005) and Kentico (2006). HubSpot (2005) is in this space as well, linking their CRM solution with content management, although that could really be considered part of a hosted solution.

Although many of these have eCommerce plugins available, many started out with a content mission and the integration of the commerce part has been a pivot. Despite current releases, these are all straying into the realm of legacy technology, with tech debt on board.

Tier Four — Prototype, Low Code and No Code Platforms

Low code and no code platforms are aimed at ‘solopreneurs’. They allow for rapid prototyping for startups. This can be necessary when despite having tight requirements documentation, you need to also try things out and iterate. They may increase collaboration between business teams and development teams. These prototypes can be developed previously in visual products such as Figma, Axure, Sketch or InVision. An actual working prototype though, may allow greater clarity. They allow a focus on the architecture and outcome rather than on the technology.

One of the best examples is Bubble.io

Low code and no code is not a great name as it makes a lot of people think that nobody had to code these in the first place, and aren’t currently coding up new features in the background. They feel like the developer version of the hosted platform, and a natural successor to services such as CodePen.

They can also be a way for developers to increase productivity by automating tedious work whilst allowing them to also write code modules.

Alongside this, connection tools such as Zapier allow integration of platform tools, code, low code and other applications to work together.

Tier Five — Static Generation and Code Frameworks

This is where we start straying into hardcore developer territory. These are projects where a developer has an IDE (Integrated Development Environment) such as Visual Studio Code and a development environment on their machine. Usually a Mac because it’s Linux based. That’s why you see all those developers rocking MacBook Air. It’s not for the shiny or the lols.

Usually a project will have a repository on BitBucket or GitHUB, with a pipeline setup to push changes to the ‘repo’ and with automation to the ‘Staging’ or ‘Live’ site. Frameworks such as Node.js (2009), Golang (2009), React (2013), Vue (2014) , Grunt (2016), Gulp (2013), Docker (2013) and other libraries form part of the project that is compiled and delivered to the web server. Sure we can also yell out, PHP (1995), .net (2002) though in heavy usage, this isn’t the most modern set of tools anymore.

This is where fully customised developments get built. The environment allows full control over the development, and the ability to control the source code, working in a team via the repository. This allows a company full ownership and control over the source code with the ability to put in place processes to minimise tech debt.

Many of the hosted solutions would have this kind of team and development structure sitting behind them.

Tier Six — Apps

Apps are built where there is some ongoing ‘utility value’. Something that you can do with the app on an ongoing basis in the context of a mobile use case. You can use it again and again, in the best case on a daily basis.

A good example is Spotify (2011). If you are listening to music on your phone or tablet, you can continue when you get home in the desktop app. There is also a web version for all devices. There is deep-linking from web to app where the app can be triggered to open and continue on with a track you have been listening to.

A plane booking app is a lesser example where the app is useful, though you might not use it every day unless you are an extreme traveller.

One of the biggest tech debt areas on apps is the integration of Google Analytics tools which can be vital for measuring conversions and interactions. Platforms such as Unity (2005), Sencha (2010), Xamarin (2012), Firebase (2011) and Flutter (2018), exist for developing apps. In this space many hosted app development tools have arisen and the same caveats apply to those, as hosted web development platforms.

Tier Seven — Insight, Data and Analytics Projects

The concept of bringing all your data together and presenting end-to-end insight is becoming a big deal. Not the dashboards we have seen for years with bar charts, pie charts and funnels only. Those have generally been coming from 3 sources. Advertising, Sales Data and Analytics. The best solutions bring these together, merging the data and allowing for ROI measurement, insight and service design.

Advertising data from agencies can include Google Marketing Platform (2018), Adobe Experience Cloud (2012) or from simpler sources such as Google Ads (2000), UTMs, Facebook (2004) and others. Customer relationship and sales data can come from products such as Salesforce (1999) and HubSpot (2005). Interaction and user data can come from Google Analytics (2005).

Data can be integrated with a Customer Data Platform such as Tealium IQ (2011), or combined in a cloud database such as BigQuery (2011).

Visualisations could then be provided in a number of business intelligence tools such as DataStudio (2016), Tableau (2005), Power BI (2011), Metabase (2014) or customised.

Tier Eight — Enterprise Systems

Although a corporate entity may have a customer-facing website with all that that entails, the service is generally delivered by a portal that needs an enterprise level framework for security (e.g. financial, online banking, customer data platforms, military).

Even in this space though, the tools may have been around a long time. For example if you were working with a legacy system it might be Java based and use J2EE an enterprise framework. This was first released in 1999, 23 years ago. COBOL code still drives a lot of old servers, particularly government. COBOL was released in 1959, 63 years ago.

There are big moves towards Tier Five frameworks for enterprise and the internet is constantly evolving in this space. There is a lot of patched code out there.

The Future

What’s the future? This analysis shows that there is a lot of legacy code and platforms out there. That means a lot of opportunity, though also a lot of tech debt for companies to get into, and hold back innovation.

Augmented Reality, Artificial Intelligence, Machine Learning, Web 3.0. I’m putting those out of scope for this article except for mentioning Unity for Augmented Reality. The categories still apply quite well to those.

Strategy Tips

Finally, some strategy tips to make the best of it, whilst the people that own the internet, take it offline and fix everything.

  1. Get specific about the requirements for your project. Have an official process for modifying the scope of a project. Try and not do this, while developers are building it. Specify the delivery milestones. Minimise meetings and feedback in the middle of the work. Make sure the documentation includes the testing plan and methodology. Every platform and system you build will have maintenance tasks specified in the requirements.
  2. Prioritise actions that focus on revenue acquisition and preservation through customer experience.
  3. Invest in prototyping and user experience (UX) architecture, to test things out before development.
  4. Make sure developers have coding standards to work to. Comment and document everything.
  5. Consider tech debt separately from project management. Time, Quality, Cost, Pick two has been the project mantra. Time mostly because of the perceived first mover advantage. Cost due to having to pay business costs until the project produces revenue. However neglecting the quality can produce debt time bombs and increase the time and cost beyond any initial savings.
  6. Over time find your greatest sources of tech debt and nail them down. Use your internal team and customer feedback loop. Have a process in place for monitoring all aspects of your business technology, with an emphasis on prioritising the tech debt backlog. Align tech debt to your businesses risk planning.
  7. Reprioritise investments in legacy systems, infrastructure and skill sets. Be flexible and use a requirements process to transition as business needs change. This often means abandoning obsolete and dysfunctional systems and processes. It means changes in customers, staff, supply chain and moving to new technology.
  8. Staged releases and minimum viable products (MVPs) may allow for customer feedback loops to form a part of product development.
  9. Keep a lid on Shadow IT where frustrated and impatient team members are tempted to deploy their own solutions outside a coherent architecture. Unmanaged tools have both security issues, create duplication in process and also break data integrity.
  10. When reading tech articles to choose platforms, also read the comments section from readers. This is where you will find some of the ‘gotchas’ with any particular platform.
  11. When using any technology from a third party (which is just about everything), familiarise yourself with the service level agreement (SLA) and how to access support. What is the cost of that support? How quickly will it be offered? What are the cost tiers? Realise support won’t generally cover anything you have customised, so who is supporting that? Is it all documented? When you use a hosted platform you might get a lot of functionality in the deal though you are putting your hands in the way that company works and its development roadmap.
  12. Will you have to add in a lot of plugins to do what you want? Is the cost of those ongoing or a one off payment?
  13. How easy is it to write custom code or customise the system? Can you build the navigation the way you want it?
  14. Is the platform you’re using geared towards your country? Particularly in regard to payment platforms and their cost? Can you display local taxes in a way that your customers expect them? Can you calculate shipping costs properly? Does the platform allow you to comply with the EU’s GDPR (General Data Protection Regulation) or CCPA (California Consumer Privacy Act) or others, if that is a factor?
  15. How does the platform protect you from scammers in the service level agreement?
  16. Pick hosting suitable for the platform tools you want to use. Virtualise where possible and use content delivery networks to ensure delivery is as near to the customer as possible. Configuration such as email needs to be considered. Don’t send bulk email on shared hosting unless you want your site shut down by the hosting provider.
  17. Sometimes platforms contain a lot of tech debt themselves and this may have a huge impact on the speed and viability of a site. Caching, compressing and minification plugins can have a positive effect on this, though this is also a function of picking a platform category suitable for your project. Legacy code can also be a security leak and a lot of sites get hacked for this reason.
  18. Consider setting up a staging environment to test changes before pushing to a live environment.
  19. If the majority of your usage is on mobile devices, particularly phones, then prioritise the experience on those devices.
  20. Making improvements on a legacy system to extend it can massively increase the cost when a new system is required.
  21. Hire the right skills. Parallel development where there are different teams (say a back end app team and front end web team) can cause issues at the handover point of the user experience.
  22. Make sure somebody is responsible for each thing. Including a great technical lead.
  23. Parts of the code may become inefficient over time and should be refactored to support future requirements. The longer refactoring is delayed, and the more code is added, the bigger the debt.
  24. Are there ongoing costs to continually promote content? Is the reach and frequency of a particular platform or advertising medium declining? Have you considered the amount of content that may have to be produced to keep up a content driven strategy? Have you factored in writing any copy? Can you structure your product content the way your business does? Can you add custom fields?
  25. What does the theme look like? Is there great photography that you will have to replace with your own? How is that going to make it look? What is the cost? Will stock imagery make your brand look ‘hokey’. What about choosing that theme, or editing it?
  26. What is the platform or payment provider’s policy on transferring your funds to you? Are they going to be held up? Can you do reverse charges? What percentage of cost is there with each transaction? Does the platform still keep a transaction fee if a sale is refunded? Can you operate multiple currencies or popular credit cards?
  27. How do you follow up on abandoned shopping carts?
  28. Are you able to sell digital products and what are the file size specifications?
  29. Do you need point of sale integration?
  30. How are you managing site membership?
  31. Where is your data sitting and on whose servers? Is this compliant with your legal responsibilities?
  32. What API (Application Programming Interface) tools are you using and what is the cost of the transfer of data from these tools?
  33. How easy is it for customers to end up with duplicate accounts due to changed or mistyped email addresses?
  34. Compile your code with the latest compatible libraries and SDKs (Software Development Kits). Do you really need every one of those libraries? Build and test reusable components.
  35. Have you considered adding analytics and reporting to your development? What reporting tools exist in your chosen platform? Are they in the tier you have subscribed to or cost extra? Do you know how you are going to bring together all the various data into a cohesive set of reports?
  36. Are you testing your site’s speed? Does the site comply with Google’s Core Vitals? Core Web Vitals are a set of targets relating to the speed, responsiveness and visual stability of a website; sites that meet them can receive preferential treatment in Google search results.

Conclusion

This article is a good primer and discussion starter on the state of technology and tech debt online. I’d love to hear comments, and I will make updates to this article if there’s something really worthwhile there.

If you would like to discuss any particular project with me personally,
I can be emailed at virgil@virgilreality.com.au

Virgil…

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Virgil Reality

Virgil Reality

Data and Analytics Professional, Media Strategist and Creative, Customer Experience Architect and Developer, Secret Trumpeter