Introduction
Quality can be measured in many different ways across industries. For instance, an article’s quality can be based on several factors, including its grammar, accuracy, clarity and conciseness. In the manufacturing industry, a product’s quality can be benchmarked against its lack of deficiencies, defects and variations. In the world of software engineering, the following definitions on software quality are considered a general standard:
- Software functional quality reflects how well it complies with or conforms to a given design, based on functional requirements or specifications. It is the degree to which the correct software was produced.
- Software structural quality refers to how it meets non-functional requirements that support the delivery of the functional requirements, such as robustness or maintainability. It has a lot more to do with the degree to which the software works as needed.
A Strong Emphasis on Quality in Software Development
For software development, quality should be a foundational step in building a company’s core product. For any company, quality must be a shared understanding across product, business and engineering. It is more important to focus on whether the customer outcome and business impact were clearly delivered upon, not just releasing bug-free code.
The general tendency across most organizations is to focus on testing at the end of the software development life cycle (SDLC), and spend significant time and effort to ensure a quality product is delivered. However, the final product quality is a sum total of the quality of deliverables in each of the upstream steps of the SDLC, which makes it hard to quantify and optimize.
Elevate The Game
eBay is consistently at the forefront of quality in execution and delivery. To take quality to the next level, we looked at the whole lifecycle of software development and identified key factors that typically have a cascading effect:
- The need to accomplish everything
- Planning and change management
- Prioritization of work
When looking at end-to-end (E2E) customer experiences, quality checks happen at different phases. eBay is a massive platform, with a very complicated mix of old and new technology stacks. Any sustainable process that spans E2E experiences needs to account for possible imperfect scenarios, but at the same time can’t come in the way of agility and speed. With the intention of elevating our standards and operationalizing a process-oriented approach, we started with the most obvious gating factors for quality, but with a twist. In a phased manner, we implemented a two-pronged approach:
- We first implemented an extremely detail-oriented Release Approval Process, which forced teams to take a step back and double-click on each step of a release life cycle, making sure all approvals/dependencies and mitigation strategies are in place.
- With the successful implementation and execution of the release approval process, we then instituted a Quarterly Release Planning (QRP) process at the start of each quarter to be methodical about each release, identify dependencies up ahead, and have an effective execution plan.
The above two processes significantly improved our agility and speed of delivery by almost 20 percent and has provided an even better standard of quality for each release. Within the next section, I dive into the release approval process and the nuances that have led to its success.
Release Approval Process
As is common across the tech sector, not every part of a company’s technology stack and features are worked on frequently, resulting in fragmented tech stacks and code bases. Despite this, we still need to deliver a quality product and a smooth customer experience every release. Simply taking a step back and looking at each release from an end-to-end perspective has helped teams deliver features with superior quality.
At eBay, we do hundreds of releases a month and many interdependent subsystems and features are well separated by interfaces and services. Given that the customer experience is a cohesive end-to-end journey, with each journey possibly split into multiple different possible scenarios, there is a possibility of a kink in the chain. When we discuss a kink in the chain of a journey, we are not always talking about bugs. A suboptimal customer journey is also considered a break in quality from a product perspective.
This thought process that we need a holistic view of every release — whether it’s a major feature release or just a bug fix — led to the inception of the release approval process. A combined look at technical design of each release, automation coverage, accessibility checks and security to name a few checkpoints formed the crux of this release approval process. None of these checkpoints were new for the teams.
As basic SDLC steps, these checkpoints are followed to this day for each feature as part of its definition of done (DoD). However, when looking at a release in its entirety, this process gave the team leads a chance to look at everything in one shot, to make sure it made sense and to check that we have not missed anything. Developed as a bottom-up organic process, based on input from the teams on what usually goes wrong in a release at ground level, this DoD became the holy grail for teams to ensure a smooth release.
The process in its entirety acts as a congregation of many different definitions of done into a single focal point: release definition of done. The harder you push, the harder the system pushes back.That truth from Peter Senge’s laws of system thinking held true during this implementation. With a solid leadership commitment to uphold quality standards, inviting engagement from all sections of the community and creating a safe environment where we acknowledge that mistakes happen. At the same time, learning and course correcting appropriately, this process became widely adopted and popular.
Looking at a release from a holistic perspective gave the team leads an opportunity to identify potential issues ahead of time. With every aspect of a release documented in one single place, it was possible to chronologically track every micro-feature in a release. We were able to identify inefficiencies related to automation and tooling that were affecting the overall agility of releases, and consequently prioritized and invested effort in these areas to further improve quality. Because of the extensive documentation in one place, this enabled business, product and engineering teams to triage any release features dependent on other features at a rate 10 times faster than before.
Results
We saw rapid, quantifiable and obvious benefits of the release DoD:
- A 25 percent increase in release volume.
- Across the board, 79 percent reduction in overall bugs.
- 35 percent reduction in prod P1 and P2.
This also ensured that dependent teams that worked on the same code base for cross-functional features adhered to a common prescribed release standard, allowing for easier testing of E2E experiences.
This is an evolving process, and there are a lot of moving parts. But what has become apparent are the significant improvements in release quality by instilling awareness of quality across the whole life cycle of a feature — from inception to how the customer’s experience is impacted. It should be noted that as engineers are becoming more confident about each release, there is less churn and stress about potential mitigation strategies needed, and there is a huge increase in the general morale of everyone involved in a feature release. Over time, seeing the benefits of this process, teams became motivated and invested in the success of this process. They proactively worked on homegrown productivity tools to automate the process and help evangelize it across the entire company.
Conclusion
What started as a gating factor at the end of the line quickly evolved into transparency and more confidence with each release. In combination with the meticulous, detail-oriented quarterly release planning (QRP) that is done at the start of every quarter, the definition of quality across the organization has changed; with every individual including product managers, engineers and quality engineers taking ownership of E2E customer experiences.
While every tech organization has its own context, challenges and processes, I am sure everyone is continuously implementing, adopting and changing to improve the quality of customer experiences. I would be very interested to hear thoughts and ideas that have worked for you.
I will leave with this saying from Aristotle that has been the cornerstone and yardstick for our focus on quality: “Quality is not an act, it is a habit.”