Excerpted from Chapter 8 of Outsource Your Own Brain by D.E. Lowery, MBA
There’s a story I love about a consultant I’ll call Talib. He built a system that scanned every news article in his clients’ industries, summarized the relevant ones, and drafted weekly intelligence briefings for each client. Beautiful system. Ran like clockwork.
Then one Tuesday, his AI summarized an article about a competitor’s product launch and included the line: “This represents a significant shift in the competitive landscape that could affect your Q4 positioning.”
It was a perfectly accurate summary. It was also something the AI had flagged three times that month about three different competitors. The signal was noise. The shift never materialized. Talib’s client wasted an hour scheduling an emergency strategy call to discuss a phantom threat.
Talib didn’t fire the system. He added a filter: the AI now compares every “significant shift” claim against the last ninety days of similar claims. If the pattern is noise — multiple similar alerts that didn’t materialize — it drops the alert and logs it for the weekly review instead of escalating immediately.
That filter cost him four hours to build. It saves his clients roughly two hours a week in unnecessary panic.
The watcher’s dilemma
Here’s the uncomfortable truth about monitoring systems: they produce too much signal and not enough meaning. Every anomaly detection system — financial, operational, competitive — generates false positives. That’s not a flaw. It’s math.
The watcher’s dilemma is that you cannot distinguish signal from noise in the moment. A single data point could be the start of a trend or a blip. The only way to know is to accumulate enough data points to see the pattern.
Most solopreneurs respond to this by either ignoring the alerts entirely (defeating the purpose) or responding to every alert immediately (burning out). Neither works.
The confidence threshold
The fix is a confidence threshold. Not every alert deserves your attention. The system needs to know when to speak up and when to stay quiet.
A practical framework:
- Below 60% confidence: Log it. Don’t surface it. Review in the weekly batch.
- 60-80% confidence: Surface it as a “worth noting” item in the daily summary. Human decides.
- Above 80% confidence: Escalate immediately. Alert the human now.
The threshold isn’t fixed — it should shift based on the cost of being wrong. If a false alarm costs you two hours of wasted work, set the bar higher. If missing a real threat costs you a client, set it lower.
Talib’s “significant shift” filter is an 80% confidence check. The AI has to demonstrate that this particular shift is different from the last three before it earns the right to interrupt the human.
The weekly noise audit
Every week, spend fifteen minutes reviewing the logged low-confidence alerts. Most of them will be noise — patterns that didn’t materialize, threats that evaporated, shifts that weren’t. But occasionally, you’ll see a pattern that did materialize — a trend that the system caught early but didn’t escalate because the confidence wasn’t high enough.
That’s your signal that the confidence model needs tuning. Lower the threshold for that specific alert type, or add a new signal that increases the confidence when it fires.
This is the loop: the system watches, the human audits, the system learns. Not in the machine-learning sense — in the “the human adjusts the rules based on observed patterns” sense. The AI is the watcher. The human is the judge. The system is what happens when they work together.
This post is based on Chapter 8 of Outsource Your Own Brain, available now on Amazon in Kindle ($6.99) and paperback ($17.99).