Why do you need Design Documents?
and how to make the most of them
There is one document that I now found crucial to have before starting any major development: a Design Document or Request for Comments (RFC).
The inception of the RFC format occurred in 1969 as part of the seminal ARPANET project.[4] Today, it is the official publication channel for the Internet Engineering Task Force (IETF), the Internet Architecture Board (IAB), and – to some extent – the global community of computer network researchers in general.
The authors of the first RFCs typewrote their work and circulated hard copies among the ARPA researchers. Unlike the modern RFCs, many of the early RFCs were actual Requests for Comments and were titled as such to avoid sounding too declarative and to encourage discussion.[6][7] The RFC leaves questions open and is written in a less formal style. This less formal style is now typical of Internet Draft documents, the precursor step before being approved as an RFC.
https://en.wikipedia.org/wiki/Request_for_Comments
Every company that I worked for have come up with their own style and flavour about how to structure Design Documents.
To me, it comes down to answer those simple (but important) questions:
What problem are you trying to solve?
How are you planning to solve it?
Why is it a good time to work on it now?
If I were to define the shortest possible process, I would start with answering the previous 3 questions.
As the process matures, you may want to expand it to include other aspects which may be relevant to be discussed at this foundational phase.
Have you thought about any problems or dependencies with other services in the company?
What is the expected traffic and load that you need to support? How do you expect it to grow over time?
Do you have any latency requirements that you need to meet?
Does it comply with security policies in the company?
How are you going to deploy it?
What metrics do you need to guarantee that you can understand what is going on while it is in production?
What happens if you need to rollback?
Creating a template that answers the previous questions is the initial step to have a useful design process that will cover most of the problems that you need to think about when designing software.
Different opinions
Over the years, I talked to several engineers who argued that it was not necessary to create this level of documentation. The most recurrent feedback was around being a waste of time which could be spent in development instead. In my opinion, I would prefer to have documented what you are planning to build before you start (so you can have a discussion about it) rather than have nothing to refer to.
Here’s a couple of examples:
You are given the task to create a new API to store customer data. This API does not exist today, and you may need to create a database to also store the customer data.
In this case, it is clear that a system design document will be very beneficial to answer the previous questions.
You are given the task to add a new endpoint into an existing customer's API to store their location data (eg: where do they live).
Do you create a new table or database?
Do you extend the current schema to accommodate the new data?
Or you rather create a new service, since the data could be treated as a separate entity?
Does the new data have any new compliance or retention policies that you need to think of?
In this case, having a system design document is also beneficial, since you can structure this data in multiple ways.
But beyond the fact of capturing the details of the work, the most important aspect is on creating conversations and knowledge transfer: an RFC without comments defies its purpose, so if you are just writing a document but you are not sharing it to get feedback on it, then you are missing the point.
To gather feedback and be challenged on why you are designing the solution in a certain way is what sparks collaboration and critical thinking. It is useful to other software engineers (who may not be familiar with the problem space) to learn from the details of the design document. This will also contribute towards knowledge sharing across the organisation.
What happens when there is no system design involved?
I have worked in teams where design documents were not created, and instead they preferred to have a bias for coding. I found this pattern more commonly in startups, where the speed to deliver to market is what matters the most.
So, what happened when development work started without any system design?
The work produced did not fix the problem that was requested initially.
When proposing a new service to replace an existing one, the present data was not analysed before designing the new system and assumptions were made. This resulted in an over-simplified architecture that did not work for the rest of the dataset, and the system was not able to be extended afterwards and had to be thrown away.
Consumers of an API were expecting a different schema from the one that was built.
Security requirements were not discussed. Services that were managing critical data were deployed to production without a proper assessment.
Performance did not meet the expectations since there were no expectations to meet.
Timelines to deliver on goals were incorrect. The team just committed to deliver unmanageable goals. This resulted in people overworking and delivering low quality just to meet deadlines.
Neither of those outcomes was a good situation to be at, and even if something was delivered I could not consider it a success.
Closing thoughts
Even with system design documents, there are still many unknowns ( you don’t know what you don’t know), but overall spending the time with the research early on pays off in the long run.
What experience do you have with Design Documents? Please leave a comment below
If you liked this post, consider subscribing, thanks!