When to Throw the Party - A Platform Engineering Problem
It's obvious that the DevOps movement destroyed the cultural phenomenon of Release Parties, where Engineering and Product finally came together after a stressful season of crunch, blame and broken promises to get drunk together after the first successful yet random build. At the same time the Ops guys shaved an ever-growing herd of yaks to deploy that particular build in production before daylight (They hadn't been invited anyways). This tradition has been replaced by a rather "boring" assumption that software development is only a success if business value has actually been delivered. Richard Campbell, an industry veteran, coined the phrase "When to throw the party" as a major question.
Fast forward almost 20 years, the question is still valid and points towards the coordination effort that makes software engineering so hard to agree on completion.
The Dilemma
Recently, we had symptoms of the same dysfunction in our platform teams. The Developer Enablement team created and released a new major version of a Helm chart used by Application teams. One of the changes introduced mandatory resource limits on pods. This was a prerequisite for our org-wide migration to AKS-managed namespaces, which was also meant to reduce resource waste.
While the release was announced, the Compute team still did not publish sizing guidelines for resource limits so Application teams got stuck between a rock and a hard place: Either risk application stability and resource efficiency by misconfiguring the limits or miss out on the other new version's features. To Application teams, this was all just 'the Platform team' — one name, two teams that apparently weren't speaking.
As there was no alignment on the incentives for the Compute and Developer Enablement teams the latter proudly announced the new version without internalizing the Compute team's cost-sensitivity, or showing empathy for the struggle Application teams now faced adopting the new version.
Everybody Loses
In result, all teams suffer:
- As the new version is not fully adopted, the old versions remain in use risking increased support efforts for the Enablement team and even requiring the deprecated version to be back-ported with new features.
- greatly exaggerated resource limits inflate the required SKU and node counts driving up costs, and limiting the Compute team's financial headroom.
- Application teams urgently need to build up SRE skills for correct workload sizing duplicating a skillset the Platform team was meant to provide centrally.
Three Ways Out
Multiple patterns and approaches exist to resolve issues like this, but I'll stick to the ones we considered and eventually applied.
First, if the Developer Enablement team had just developed and integrated the new version without releasing in public, the team could have reduced the WIP and met their goals. Applying the Dark Launch pattern is usually attractive in Release Management for Web Applications, but not a good fit in Platform Engineering as coupling with the customer and peers is tighter, some users will find and adopt the new version anyways, and bugs will go unnoticed surfacing weeks and months after engineers switched tasks and context.
As mid-term solution, we advised product owners to label backlog items that require coordination and conduct refinement collaboratively with all teams involved. This is surely no silver bullet as it increases collaboration effort, but internalizes the perception of being one Platform team.
Finally, Team Topologies has a name for this: when a value stream is split across team boundaries, coordination overhead becomes a permanent tax, not a temporary cost. The fix isn't more refinement meetings — it's collapsing chart ownership and compute ownership into the same group, so the coordination problem disappears because there's nothing left to coordinate.
So, When Do We Party?
But the hardest question remains: When to throw the party then? In fact, more parties are required. In this case, the first party needs to be celebrated after the first Application team adopted the new chart configuring sensible resource limits as this proves not only reasonably good, self-sufficient documentation, but also that the new chart version provides features the business actually wants to use.
The second party comes after joint refinement catches a required synchronization point before things break — definitely a moment to catch a breath, open a soda, and overstate the risk the team just averted by... being an actual team. This is the mid-term fix from earlier, paying off in real time.
But what about the re-org? Shouldn't that be a reason to party in itself? I'd argue against it as re-orgs are oversold on projected happy-path outcomes making the usual engineer suspicious. After a successful re-org, nobody will hand you a medal, as smooth flow rarely gets attention.
Lastly, I am not your dad, party as much and as hard as you'd like. Work is hard and time-consuming anyways, so celebrating wins as little as they might seem, is important to keep your sanity.