On August 1, 2012, Knight Capital lost $460 million in about 45 minutes because of a switch it had stopped using nine years earlier.
The details come from the SEC’s 2013 order against Knight, and they read less like a trading story than a delivery story. In 2003, Knight stopped using an order routing function called Power Peg. It left the code on its production servers. In 2005, engineers moved a piece of logic that Power Peg depended on, and nobody retested Power Peg, because nobody used it. In 2012, a new feature for the NYSE Retail Liquidity Program reused the flag that had once switched Power Peg on. A technician copied the new code to seven of eight servers. On the eighth, the old flag woke up the old code, which no longer knew when an order was filled. It kept sending orders: 4 million executions in 154 stocks, more than 397 million shares, before anyone stopped it.
Every step of that chain was a small, reasonable decision by someone who had other priorities. Leaving the code in was cheaper than taking it out. Not retesting dead code was efficient. Reusing an existing flag was tidy. That is exactly why feature flag cleanup is a product problem and not just an engineering chore. The decisions that create flag debt are made on delivery timelines, and delivery timelines belong to the product manager as much as to anyone.
A flag is a decision you haven’t finished making
Feature flags are one of the best things to happen to delivery in the last fifteen years. They let a team deploy code without releasing it, roll out to 1 percent of users before 100, and kill a broken feature without a rollback. We’ve covered why separating deploy from release changes the risk math, and none of that is in dispute here.
The trouble is what happens after the rollout reaches 100 percent. At that point the flag has done its job. The experiment is decided, the launch is over, the team has moved on to the next thing. The flag stays. So do both code paths behind it: the new one everyone uses and the old one nobody does.
Pete Hodgson’s widely cited feature toggles article on Martin Fowler’s site puts it plainly: savvy teams treat the toggles in their codebase “as inventory which comes with a carrying cost” and work to keep that inventory as low as possible. Inventory is the right word. A warehouse full of parts you will never ship still costs rent, still needs counting, and still gets in the way of the parts you do need.
Here is the reframe that helped the teams I’ve worked with: a live flag means a decision is still open. Either you are still deciding whether the feature should exist, or you have decided and haven’t told the code yet. The second state is the dangerous one, because it looks finished on the roadmap and isn’t finished in production.
How big the pile gets
This is not a problem of a few forgotten switches. When Uber introduced its automated cleanup tool, Piranha, its engineering team wrote that it had used the tool to remove around two thousand stale feature flags and their related code from its mobile apps. The team described those flags as technical debt “making it difficult for developers to work on the codebase,” and noted that removing them by hand was time engineers could not spend on new features. The work was serious enough to become an ICSE 2020 paper on reducing feature flag debt at Uber.
Google’s Chromium codebase is another public example. A 2023 study of toggle usage in Chromium counted 3,100 toggles across 38 components and catalogued the patterns that turn them into code smells: toggles nested inside other toggles, toggles spread across components, and dead toggles that no longer control anything but still sit in the code.
Those are two of the best-resourced engineering organizations in the world. If they accumulate flags by the thousand, your team of eight is accumulating them too. You just haven’t counted.
Why the cost stays invisible to product
Flag debt hides from product managers for three reasons, and each one points at something a PM can actually change.
The cost lands on someone else’s sprint. The flag was added during your feature’s delivery, when the deadline was close and cleanup felt optional. The cost shows up months later, when a different engineer working on a different feature has to reason about a branch that might or might not be live. That engineer’s estimate goes up by a day, and nobody connects the day to your launch.
Testing multiplies quietly. Every independent flag doubles the number of possible configurations a piece of code can run in. Ten flags touching the same area means over a thousand combinations. No team tests them all. They test the one that’s on in production and hope the others stay off. Knight’s Power Peg was precisely an untested combination that someone assumed would never run.
“Done” was defined at launch. Most teams mark a feature complete when it reaches 100 percent of users. The flag removal ticket, if it exists at all, lands in the backlog with no deadline and no owner, below every customer request. It will lose that prioritization fight every week, forever, because nothing about it is urgent until the day it is catastrophic.
I saw a smaller version of this in IT operations at a large telecom. We had a configuration switch left over from a platform migration, kept “just in case” we needed to fall back. Years later, after the fallback environment no longer existed, a routine change script flipped a related setting and pointed a batch of service orders toward a system that wasn’t there. Nothing exploded. The orders simply sat, and it took most of a day to work out why, because the two people who remembered what that switch did had both moved to other teams. The cost wasn’t the outage. It was that nobody could reason about our own system anymore.
Feature flag cleanup belongs in the definition of done
Engineers know flags are debt. What they usually lack is permission to spend time removing them, and that permission comes from whoever controls the definition of done. That is often you. A few practices make the difference.
Give every flag a type and an expiry when it’s created. Hodgson’s article separates release toggles (temporary, days to weeks), experiment toggles (live until the test is decided), ops toggles such as kill switches (sometimes long-lived by design) and permission toggles (often permanent). Only the last two have any business living past a launch. If a release flag has no expiry date written down on the day it’s created, assume it will live forever.
Create the removal ticket in the same breath as the flag. Some teams in Hodgson’s account add a removal task to the backlog the moment a release toggle is introduced. Better still, make removal part of the feature’s acceptance criteria, so the feature is not done until the old path is deleted. A feature at 100 percent with its flag still in place is the 90 percent done trap with a nicer dashboard.
Let the build enforce it. The same article describes “time bombs”: tests that fail once a flag outlives its expiry date. It sounds aggressive. It’s the only mechanism I’ve seen that survives a busy quarter, because it converts a forgotten decision into a red build that someone has to look at.
Cap the inventory. One Lean approach Hodgson mentions is a hard limit on how many flags a system may have at any time. Want a new one? Retire an old one first. This turns cleanup from a virtuous extra into a precondition for the next launch, which is the only kind of priority that actually wins backlog fights.
Never reuse a flag. Knight’s new feature borrowed an old flag’s name. Retired flags should be deleted along with the code they guarded, so the name cannot accidentally mean two things on two servers. Uber’s approach of treating a flag as stale after a set period without changes (its example was 8 weeks) is a useful default trigger for a review.
Review the list where you review the roadmap. Once a quarter, put the count of live flags, and the age of the oldest release flag, in front of the same people who review delivery. It takes five minutes. The first time you do it, someone will be surprised by the number, and that surprise is the point.
The question to ask at your next launch review
When a feature hits 100 percent, most launch conversations ask whether it worked. Add one more question: “When does the flag come out, and who is taking it out?”
If the answer is a date and a name, the launch is finished. If the answer is a shrug, you haven’t shipped a feature. You’ve shipped a feature plus a switch that someone, years from now, will flip without knowing what it does. Knight’s engineers in 2012 were not careless people. They were working in a system whose earlier owners had left a decision open, and the bill arrived nine years later, in 45 minutes, for $460 million.
The cleanup is cheap when you do it in the week after launch, while the people who built it still remember why. It only gets more expensive from there. That’s the same logic behind treating technical debt as a product decision: someone is going to pay, and the only choice you get is when.