Why Is Software Still Built From Scratch?

A few months ago, I'd visited the South Indian temple town of Tirupati with my family. On the way, whenever the train stopped at a station, the engine was switched off. But the air-conditioning still worked because it was powered by huge battery packs installed in the AC coach. Coincidentally, the manufacturer of these battery packs is located close to Tirupati. I know this company from the time it purchased my employer's ERP in the early 2000s. Through the years, I've stayed in touch with the ex-CEO of the unit. 

I met him during my latest trip and introduced him to my family as the guy who "built the electronic circuit that manages the battery packs on AC coaches of Indian Railways trains".

He corrected me, saying, "I assembled the circuit using available integrated circuits without building every part of it."

Yes, indeed. Since times immemorial, automobiles, printed circuit boards and many other products are assembled from standard components, which can be sourced from catalogs.

But not software.

I joined the software industry two decades ago. From then to today, I've been hearing of technologies like SOA, CBSA and microservices that purportedly allow programmers to build systems by assembling prebuilt functionality.

But that vision has still not turned into reality.

Once upon a time, developers had to build software systems from scratch perhaps because there was no prebuilt functionality.

Then, it's quite likely that developers built reusable components by using technologies like SOA. But, despite the existence of methodologies like CBSA, development teams still had to build software from scratch because they didn't know about the existence of reusable components.

Then came innovations like open source repositories and open API marketplaces.

For the uninitiated, an open source repository (e.g GitHub) contains prebuilt source code for components that can be used by programmers to develop their own systems; and an open API marketplace (e.g. Algorithmia) expose readymade functionality that can be accessed by developers from their systems via application programming interfaces.

In theory, if you want to develop a new system, you should be able to review the catalog on these platforms, pick and choose the component or API you need, and integrate the prebuilt functionality into your new system.

In practice, this has worked reasonably well in building software pilots / proofs of concepts / MVPs.

However, when it comes to enterprise systems, these platforms haven't made a big dent in the traditional way of building software systems from scratch. I'll outline the key hurdles of changing the time-worn development methodology by continuing to use the PCB analogy:

1. A 12μF (twelve microfarad) capacitor will work on any PCB. Whereas, since software is heterogeneous, a .NET component won't work in a Java system.
2. You can't manufacture a capacitor ab initio even if you wanted to - at least not within the timescales of most PCB development projects. Whereas, most development teams believe that they can develop any piece of software from the ground-up within software project timescales.
3. You use a capacitor as it is. Whereas developers inevitably need to make some changes to borrowed code before being able to integrate them into their systems. According to open source foundation rules, you're required to share the enhanced / modified code back with the community. That can be tricky if it contains a company's secret sauce. You either flout OSF rules - à la  the investment bank described in Michael Lewis's book Flash Boys - or eschew open source code altogether.
4. You can buy a capacitor. But you can only license an API, which is not the same as buying it. The license is subject to the Terms of Service of the API owner ("OEM"). Now, many of these TOS are ambiguous and easy to infringe without any wrongful intent. The deadpool is littered with startups that got on the wrong side of the TOS of an 800-pound gorilla's API.
5. Once you plug a capacitor into your PCB, it will work the same way throughout its lifetime. Whereas an API won't. When its OEM releases its next version, you'll need to modify your software such that it works with the updated API. We faced a costly change to our HEATMAP360 social intelligence platform when Twitter changed its API specs.
6. If the manufacturer of a capacitor (also "OEM") folds up, your PCB will still work. But, if the API's OEM dies, your system will stop working and you'll need to find an alternative source for the said functionality. As illustrated by the examples in Does Cloud Increase Vendor Risk? (hyperlink removed to comply with Finextra Community Rules but this post should appear on top of Google Search results when searched by its title), the death of a few API OEMs has caused existential crises for many software systems and their owners.
7. Then there's the big elephant in the room that no one talks about: Programmers don't like working on code written by anyone other than themselves.

As a result of these challenges, it has been hard to develop enterprise-grade software systems by assembling prebuilt functionality.

That's not to say it can't be done.

But cracking the Holy Grail of software engineering will require structural changes in the way software is packaged and sold. The purchase process for software should mimic that of electronic components whereby customers can:

• discover software components in an open catalog
• find out their specifications and select which components they need to assemble their system with
• buy the selected components outright without onerous licensing terms or dependency upon any form of first- or third-party IP
• be assured of getting the same functionality throughout the life of the software, and
• modify components without having to share the source code with external entities.

If you're wondering "what's in this" for a software product vendor, you're not alone. Actually, this model is more closely aligned with IT services. More on that in a follow-on post.