In anticipation of Mark Richard's keynote, "The Intersection of Architecture and Implementation", Dehla Sokenou and Lukas Zühl (WPS - Workplace Solutions) had a series of questions about his views and experiences on the alignment of software architecture and implementation.
You state that software architecture and implementation are often treated as two separate things. Is that really still the case in agile software development and if so, how often have you experienced it?
Unfortunately, the separation between architecture and implementation still exists in agile software development, and in some cases, the divide is even greater due to the perception that architecture isn’t necessary in Agile. I’ve experienced this issue on almost half of the projects I’ve been involved with. Turns out, the involvement of an architect in agile software development is even more important than other software development methodologies because architecture is an iterative process. The more we learn about the problem, the more we change the architecture to accommodate the new information. This requires an architect to be involved throughout the entire lifecycle of a project.
For example, let’s say you are building an order entry system, and you start out by creating a single payment service that accepts and processes credit card payments. Later down the road, a new requirement comes in to support PayPal as payment, then WorldPay as an additional payment type. The payment service starts getting bigger and more difficult to maintain. You hear that gift card and reward point functionality is coming up. So, as an architect, you decide to break apart the payment service into separate services, one for each payment type. This is an architectural refactoring to help support further requirements and maintenance. It’s architectural because you are changing the structure of the system - new deployment units, new data schemas in our databases, possibly a new orchestration service to manage multiple payments for a single order, and code to support distributed transactions.
You see, in agile methodologies the architecture evolves just the same as the implementation (source code and functionality) does. We learn more about the system in each iteration and make adjustments to accommodate the new requirements. Therefore, although they are two separate things, they must be kept in sync in order to make software work.
It's common for projects to undergo changes and refinements over time. How do you recommend managing the alignment between architecture and implementation when the software's requirements evolve?
All architecture is iterative and must change as the business and technology change. However, since code changes at a much faster rate than the architecture, they both need to stay in alignment to support the things that are important to the business (characteristics like scalability, responsiveness, time-to-market, and so on). One effective way of ensuring alignment between the architecture and implementation is through the use of fitness functions. A fitness function is an objective integrity assessment of some sort of architectural characteristic. For example, you might write a fitness function to capture and measure the overall responsiveness of the system (or one specific part of a system). As changes are being made to the system through implementation, you can run the fitness function to make sure your changes haven’t impacted the required responsiveness requirements. The same is true with architecture. Changing a protocol, breaking apart or consolidating services, changing a contract or interface might impact responsiveness. Fitness functions can help ensure that alignment.
In addition to alignment based on operational characteristics, there’s structural alignment as well. Fitness functions can be used to ensure that certain architectural constraints (such as two components that shouldn’t communicate with each other) are met. I’ll be talking a lot about these in my upcoming keynote session.
Is software architecture relevant at all or should we just go with the flow and continuously react on what is implemented?
I like to answer this question by asking if you would be okay with allowing a bricklayer or electrician to just wing it and “go with the flow” and start constructing your new house from the ground up. Most people would say no. Buildings require an architecture so they don’t come crashing down due to a design flaw or load problem. The same is true with software. Now, I’m not saying that you require a complete architecture before you can start developing software. What I am saying is that you need to know what it is you are building before you start. For example, knowing the architectural characteristics that are critical to the success of the system and what some of the major building blocks are before beginning implementation is important. Are you building a distributed system like microservices or a monolithic system like a modular monolith? How should the source code be structured?
As I mentioned before, all architecture is iterative. The architecture must evolve as the requirements evolve. Therefore, it’s normal for the architecture to change during the course of a project. Even today, this is a foreign concept for a lot of people, particularly project managers and business stakeholders. Occasionally people in these roles will see constant architecture change as a problem; that you don’t understand what you are doing or you are constantly getting it wrong. However, the exact opposite is true—constant change to the architecture is a sign that it is conforming to new information and new requirements and that it is in sync with implementation.
Could you elaborate on a scenario where a software project failed to maintain the alignment between architecture and implementation throughout its lifecycle? What were the key factors that contributed to this situation, and what lessons can other projects learn from it?
A common scenario I’ve experienced numerous times throughout my career where a project failed to maintain an alignment between architecture and implementation is when the implementation doesn’t adhere to the structural constraints of the architecture. In this case, we had a large traditional monolithic n-tiered layered architecture with the classic presentation layer, business layer, shared services layer, and finally the persistence layer. One of the architectural constraints of the system was that the presentation layer could not access the database directly—it had to first call the business layer to get to the database. The same was true with the business layer—it had to go through the persistence layer to access or update data. This architectural constraint was in place to control constant change to the underlying relational database. By isolating database access only to the persistence layer, database changes were easier and faster because the isolated (and dedicated) persistence logic was the only thing that had to change.
After the system was released into production, the misalignment between the architecture and implementation became very apparent. As expected, database table structure changes occurred at an alarming rate. The architecture supported this change due to having strict layers of separation, but unfortunately, the implementation did not adhere to those layers. Presentation code was calling the persistence layer (and in some cases the database directly) to retrieve data because it was faster, and code in the business layer was making database calls without going through the persistence layer. As a result, every database structure change permeated throughout the entire system, including the user interface and presentation code. It was a mess, and database changes became a major effort.
The lesson learned from this disaster was “don’t assume that the implementation and architecture will always be aligned” and “don’t assume architectural conditions and constraints will always be followed”. Implementation can (and usually does) deviate from the architecture and architectural constraints. It’s up to the software architect to ensure alignment of these two things at all times. Many times a development team is unaware of the architecture and/or architectural constraints, other times they are simply ignored in favor of a “better” approach. Lack of collaboration, communication, and automated governance were mostly to blame for this failure.
I’ll be covering a lot of actual real-world scenarios like this in my keynote session, as well as techniques to make sure this doesn’t happen to you.
Are there emerging trends or methodologies that you believe will play a significant role in strengthening the alignment between architecture and implementation in the future?
There are several emerging trends and practices that help aid in the alignment of architecture and implementation to achieve a common goal. It would take too long to dive into the details of each of these trends, so I’ll talk about what these trends are and where to go to get more information about them.
The first trend, of course, is fitness functions. Fitness functions are gaining in popularity and are becoming a core part of any ecosystem. You can get more information about what fitness functions are and how they work by referring to the books “Building Evolutionary Architecture, 2nd Edition” (O’Reilly) and “Software Architecture: The Hard Parts (O’Reilly).
The second important trend to help in the alignment of architecture and implementation is the concept of an architectural quantum. An architecture quantum (borrowing from the physics term) is the smallest part of a system that can completely stand alone—from the user interface all the way down to the database (and everything in-between). This concept is also introduced and described in detail in the books “Building Evolutionary Architecture, 2nd Edition” (O’Reilly) and “Software Architecture: The Hard Parts (O’Reilly). This helps maintain alignment by segmenting the system into separate and distinct parts. Admittedly, this really only applies to distributed architectures as monolithic (single-deployment applications) represent a single architectural quantum.
The use of AI in software architecture is still questionable and in its infancy, but AI is undoubtedly a trend in the software field. The hope is that someday AI will be able to understand architecture, understand the underlying implementation, and look for deviations between the two. This is an exciting venture, one that might show some promise sometime down the road.
Sponsoring a conference is a terrific way to support and connect with our global community of software architects.
