From Chaos to Clarity: The Art of Processes in TPM and AI

In the dynamic world of tech, effective processes can be the difference between chaos and clarity. Join me as I share my hard-earned lessons on incident management, SLOs, quality gates, and the delicate balance of governance and speed in the realm of TPM and AI.

Navigating Crisis: A Defining Moment

It was a Friday afternoon, the kind where the weekend feels just a heartbeat away. I was huddled in a conference room with my team, a mix of engineers and product managers, staring at a wall of sticky notes that felt more like a chaotic art installation than a roadmap to success. We had just experienced a significant outage in our AI service, and I could sense the tension in the air. This was not just another incident; this was a defining moment for us as a team—a moment where our processes would either shine or crumble.

As a Technical Program Manager (TPM) in a big-tech environment, I’ve often found myself in the trenches, battling both the complexities of technology and the intricacies of human behavior. One lesson I’ve learned the hard way is that effective processes can be the lifeblood of a project, especially when it comes to incident management, SLO/SLA hygiene, release trains, and quality gates.

Let’s start with incident management—a process that can make or break our ability to learn and adapt. I vividly remember the first time I facilitated a blameless postmortem. We had just recovered from an outage that impacted thousands of users, and I was terrified of the finger-pointing that usually followed. Instead, I introduced a framework that shifted our focus from blame to understanding. We dissected the incident collaboratively, identifying root causes and documenting lessons learned. This approach not only fostered a culture of psychological safety but also led to actionable insights that prevented future issues. The takeaway? Embrace a culture of blameless postmortems; they are a powerful tool for continuous improvement, especially in the fast-paced world of AI.

Next, let’s talk about SLOs and SLAs. In my experience, these metrics are not just numbers on a dashboard; they are commitments to our users and a reflection of our operational health. I once worked on a project where we had set overly ambitious SLOs, thinking that would push us to excel. Instead, we found ourselves in a constant state of stress, chasing metrics rather than improving our service. By involving the engineering team in the SLO discussions, we recalibrated our expectations to be both realistic and aspirational. The result? A more motivated team and a product that truly met user needs. Remember, SLOs should be data-informed and adaptable, not burdensome chains.

Release trains and quality gates can often feel like bureaucracy in disguise. I’ve been part of environments where a release was held up for weeks due to an exhaustive checklist that did little to ensure quality but a lot to frustrate the team. I learned that a streamlined release process, with clear quality gates that are meaningful—rather than merely procedural—can result in faster delivery with less friction. In one project, we implemented a two-way communication channel between QA and engineering, allowing for quick feedback cycles and ensuring that quality wasn’t just a checkbox but a shared responsibility. This shift not only improved our release cadence but also enhanced team morale.

Design and PRD review rituals are another critical aspect of our processes that deserve attention. Early in my TPM career, I witnessed countless hours of debate over design documents that often resulted in more confusion than clarity. I realized that the key was not just to have a review ritual, but to ensure it was a space for constructive feedback and creative collaboration. We adopted a 'design thinking' approach, encouraging diverse perspectives and focusing on user outcomes rather than technical specifications. This not only improved our product quality but also empowered team members to voice their ideas freely.

As we dive deeper into the world of AI, we encounter unique challenges that require us to balance governance with speed. The rapid pace of AI development can create an environment where teams feel pressured to deliver without adequate oversight. However, I’ve learned that a healthy process doesn’t stifle innovation; rather, it provides a framework for it. We implemented lightweight governance processes that allowed teams to move quickly while still adhering to necessary checks. For instance, we established a 'fast track' review for AI models that met certain criteria, allowing for rapid iteration while ensuring quality.

Of course, we must be wary of anti-patterns that can arise in our processes. Bureaucracy can creep in, turning our well-intentioned frameworks into cumbersome chains. Cargo cult behaviors can emerge, where teams blindly follow processes without understanding their purpose. I’ve seen teams become so enamored with the rituals of Agile that they forget the principles behind them. It’s crucial to foster a culture of reflection and adaptation, where processes evolve based on feedback and outcomes.

As I reflect on these experiences, I’m reminded that the heart of effective processes lies not in rigid structures, but in the flexibility to adapt and learn.

Empowering Teams Through Harmonious Processes

We are not just managing projects; we are orchestrating a symphony of human talent and technological prowess. In this dance between governance and speed, let’s strive for processes that empower rather than constrain, that encourage learning rather than blame, and that celebrate innovation rather than stifle it.

In the end, it’s about finding the right balance. As TPMs, our role is to navigate these waters with humility, humor, and a relentless focus on bringing out the best in our teams and products. After all, in the ever-evolving landscape of technology and AI, the only constant is change—and our processes should reflect that.