How to Reduce Alert Fatigue: The Permission Problem

What's in this guide

• The four numbers that tell you how bad your alerting is, and why "alerts per week" is not one of them
• Why threshold tuning and AI triage are the wrong first move, and the two cases where they are not
• Why alert audits produce a list and then nothing happens
• An eight-step removal framework built around permission rather than analysis
• Where tooling genuinely helps, and the three things it cannot fix
• The trade-offs, including when this advice is wrong

Alert fatigue, briefly

Alert fatigue is the state where responders stop reacting properly to alerts because there have been too many of them for too long. The full definition, the symptoms and the standard causes are in the alert fatigue glossary entry, and the wider discipline it sits inside is covered under alert management.

One thing does need saying here, because the rest of this guide depends on it. Alert fatigue is a stock, not a flow. It is the accumulated result of every alert rule anyone ever added, minus every rule anyone ever removed. The second number is almost always zero. That is the whole problem in one sentence, and it is why tuning the newest noisy alert never seems to help: you are adjusting the flow while the stock keeps growing.

The four numbers that tell you how bad it is

Most teams measure alert volume, which is the least useful number available. Volume tells you the size of the pile. It does not tell you what fraction of the pile deserved to exist, which is the only question that leads anywhere.

Here are four numbers worth having instead. You can get all of them from your alerting tool's history in an afternoon.

Actionability rate is the headline number and it is usually shocking the first time. Teams that have never measured it tend to land somewhere between 5 and 15%. If yours comes in under 20%, the alerting is actively training people to ignore it, which is worse than having no alerting at all, because no alerting at least does not create false confidence.

Rule concentration is the number that decides your strategy, and almost nobody calculates it. Run it before you do anything else. In most teams the ten noisiest rules produce somewhere between 60 and 80% of all alerts. If that is you, this is not a culture problem or a tooling problem, it is ten rules, and you can be most of the way out of it in a week. If your ten worst rules produce 25% of volume, the noise is spread across hundreds of rules and you have a genuinely structural problem that needs the full framework below.

Repeat rate separates the two kinds of fix. A high repeat rate means one failure is producing many notifications, which is a compression problem that grouping incidents to reduce noise solves without removing a single rule. A low repeat rate with a low actionability rate means the rules themselves are wrong, and no amount of compression will save you.

Measure these four before you change anything. Not because measurement is virtuous, but because the fix for a concentration problem and the fix for a structural problem are different, and guessing wrong costs you a quarter.

Why threshold tuning and AI triage are the wrong first move

These are the two things every guide recommends, they are both real levers, and neither is the biggest one available to you.

The arithmetic that decides it

Suppose 500 alerts a week, actionability rate 10%. Fifty alerts mattered, 450 did not.

Tune your thresholds aggressively and you might cut the 450 to 300. That is a genuine improvement and it took weeks of careful work per rule, because tuning is per-rule work and there is no bulk operation for judgement. You now have 350 alerts a week at a 14% actionability rate, and the same number of rules, all of which will drift again as traffic grows.

Delete the 40 rules that produced those 450 alerts and never once produced an action, and you have 50 alerts a week at close to 100%. It took an afternoon. Nothing drifts, because the rules are gone.

The reason everyone recommends tuning anyway is not that it works better. It is that tuning is permitted and deleting is not. Tuning a threshold is a reversible config change nobody will ever ask you about. Deleting a rule is a decision with your name on it.

Suppression keeps the rule alive

Snoozing, muting, routing to a low-priority channel and "we filter those in Slack" are all suppression. Suppression is genuinely useful, and it is what you reach for when you cannot get permission to delete.

The catch is that a suppressed rule is still there. It still needs maintaining, it still fires into something, it still appears in the config when the next engineer tries to understand the alerting, and the suppression itself becomes load-bearing infrastructure that nobody documented. Six months later somebody removes the filter during a Slack migration and the noise comes back with interest.

Use suppression deliberately, for the rules you have decided to keep but not to page on. Do not use it as a substitute for a decision you were not allowed to make.

There is a legitimate version of this, and it is worth being precise about where the line sits. Smart alert handling covers the cases where the rule should survive but the notification should not: temporarily snoozing an integration that is misbehaving rather than deleting the integration, escalating by condition so weekend coverage differs from weekday, and routing by attribute so the alert reaches the team that owns it. Those are decisions about delivery. They are not decisions about whether the rule should exist, and the mistake is using the first to avoid making the second.

What AI does and does not move

AI-assisted triage genuinely helps with correlation, summarisation and first-pass enrichment. It can tell you that these fourteen alerts are one event, and it can draft the context so the responder starts from a paragraph instead of a payload.

What it does not do is decide that a rule should not exist. That decision requires knowing what your business considers an acceptable failure, and there is no telemetry for that. Pointing a model at a firehose of worthless alerts produces well-summarised worthless alerts, delivered faster.

There is also a specific trap here. AI triage makes noise cheaper to tolerate, which removes the pressure that would eventually have forced someone to fix it. A team that can survive 500 alerts a week will keep having 500 alerts a week, and the underlying stock keeps growing underneath a layer that makes it feel manageable. That is worth knowing before you buy it, not after.

When the standard advice is right

Two cases, and they are common enough to name.

Your repeat rate is high and your rules are basically sound. One failure produces forty notifications, but the forty rules are all things you genuinely want to know about. This is a compression problem. Deduplication and grouping fix it directly and you do not need to remove anything. Go straight to step 7 of the framework and skip the rest.

You are brand new and the alerting was inherited. If you cannot yet tell which rules matter, tuning and observing is the correct way to build that knowledge. Spend a month watching. Then come back and start deleting, because the knowledge was the point.

Why alert audits produce a list and then nothing happens

Every team that has an alert fatigue problem has, at some point, run an audit. Someone exported the rules, marked the bad ones, and presented the findings. The findings were correct. Some of them may even have been actioned. Six months later the volume is back where it started.

This happens for a reason that has nothing to do with analysis quality.

Deleting an alert has an asymmetric blame profile. If you delete a rule and nothing happens, which is the overwhelmingly likely outcome, you receive nothing: no credit, no acknowledgement, no line in a review. If you delete a rule and six weeks later the one incident it would have caught happens, the postmortem contains a sentence with your name in it. The expected value of deleting an alert, calculated by the person who has to do it, is negative.

Keeping an alert has no cost profile at all. Nobody is ever blamed for a rule that fires uselessly. The cost is real, but it is diffuse, delayed and paid by whoever happens to be on call, which is frequently not the person who would have to take the risk of removal.

So the rational individual behaviour is to leave everything alone, and the aggregate result is a monitoring configuration that only ever grows. Every team I have seen with a serious alert fatigue problem has arrived there through a series of individually reasonable decisions by people acting sensibly given the incentives in front of them.

This is why the fix has to be structural rather than analytical. You are not trying to find the bad alerts, you already know which ones they are. You are trying to build a process where removing one is not a personal risk. Everything below is in service of that.

A blameless culture is the precondition. If your postmortems assign fault, none of this will work, because the asymmetry you are trying to remove is the one your incident process keeps recreating.

The removal framework

Eight steps, in order. Steps 1 to 5 are about permission. Steps 6 to 8 are about compressing what survives. Most teams try to start at step 7, which is why most teams are still tired.

Step 1. Build the inventory from what fired, not from the config

Export every alert that actually fired in the last 90 days, not the list of rules that exist. These are very different documents. The config contains rules that have not fired since 2023 and cost you nothing, and it omits the fact that one rule fired 4,000 times.

Group by rule. Sort by count descending. Calculate rule concentration from the top of that list. You now have the only document that matters, and it took twenty minutes.

Step 2. Score every rule on one question

For each rule, ask: when this fired, did a human do something they would not otherwise have done?

Not "was it interesting." Not "could it theoretically matter." Did somebody act. You can usually answer this from the incident record: did the alert produce an acknowledgement followed by a change, or did it produce an acknowledgement followed by silence.

One question, because a scoring rubric with six dimensions is how audits turn into projects and projects turn into nothing. If you cannot answer it from the data, ask the person who was on call. They will remember.

Step 3. Sort into four buckets and be ruthless about bucket sizes

If your delete bucket is under 20%, you have not been honest. Run it past the person who carried the pager last month and watch the bucket grow. The failure mode of this exercise is a delete bucket of four rules and a tune bucket of two hundred, which is the same as doing nothing but takes a quarter.

Demote is where most of the argument goes, and it is usually the right answer for the rules people feel strongly about. "We need to know if disk usage crosses 80%" is true, and it is not a reason to wake anyone. Demotion lets you win that argument without asking anyone to accept blindness.

Step 4. Make removal reversible, and say so in writing

This is the step that makes the rest work, and it is the one nobody writes about.

Before you delete anything, write down and circulate three things:

1. Deleted rules are kept, not destroyed. Commit the removed definitions to a file in version control. Restoring one is a revert, and the restore takes minutes. If your alerting is defined as code this is free; if it is not, a text file is fine.
2. Deletion is a team decision, recorded under the team's name. Not an individual's. The record says "the platform team removed these 34 rules on 14 October following the Q3 alert review," and it links to the review. No individual name appears anywhere in it.
3. If a removed rule would have caught a later incident, that is an expected outcome of the process, not a failure by a person. Write this sentence down in advance, before you need it, and get whoever runs postmortems to agree to it in advance too. Its entire purpose is to exist before the incident rather than be argued for afterwards.

Point three is the load-bearing one. You are pre-committing to the fact that this process will occasionally be wrong, and deciding now who owns that: the process, not the person. If you cannot get agreement on that sentence, stop here. The rest of the framework will not survive its first bad outcome, and you are better off knowing that before you delete 34 rules than after.

Step 5. Give every surviving rule an owner and an expiry date

Every rule that survives gets two new fields.

An owning team. Not a person, a team, because people leave and rules outlive them. The owning team is the only group who needs to agree to remove it later, which turns the next removal from an organisation-wide negotiation into a standup item.

An expiry date, twelve months out. This is the single highest-leverage change in the framework, and it is close to free.

An expiring rule inverts the default. Today, a rule persists unless someone takes the risk of arguing for its removal. With an expiry date, a rule persists only if someone actively renews it, which means the burden of justification moves from the person who wants it gone to the person who wants it kept. Nobody has to be brave. They just have to not renew.

In practice renewal is fast, because most rules are obviously worth keeping and get renewed in a batch. The ones that generate hesitation are exactly the ones worth discussing, and the hesitation itself is the signal. You have replaced an audit that needs a champion with a calendar event that runs itself.

Step 6. Separate waking someone from telling someone

Everything that survived now gets sorted into two classes: things worth interrupting a human for, and things worth recording. Most alerting systems do not distinguish these, which is why they default to interrupting.

This is what severity levels are for, and the common failure is defining them against internal metrics rather than user-facing impact. "Database CPU above 90%" is not a severity, it is a reading. "Checkout is failing for a measurable share of customers" is a severity. Note also that severity and priority are not the same thing, and conflating them is how everything ends up at the top tier.

Two mechanisms make the split real rather than decorative:

• An error budget tied to a service level objective. If the error rate is inside budget, it is not a page, it is a Tuesday. This gives you a principled and pre-agreed answer to "but what if it matters," which is otherwise the argument that promotes everything back to urgent.
• An escalation policy with different paths per tier, so demotion has somewhere to go. A demoted alert that still pings the on-call phone has not been demoted. We wrote about the specific case of stopping auto-escalations for minor incidents separately, because it is the most common place this breaks.

Expect severity inflation within two quarters. Everything becomes top-tier once top-tier is the thing that gets attention. Review declared severities monthly against what actually happened and correct gently and publicly, the same discipline that keeps an escalation matrix honest.

Step 7. Compress what is left

Only now is compression the right move, because you are compressing signal rather than preserving noise in a tidier format.

Deduplication collapses repeats of the same alert into one incident. Alert grouping clusters different symptoms of one failure: when a database saturates, the latency alert, the timeout alert and the error-rate alert are one event with three faces, and they should arrive as one incident carrying all three. The mechanics of both, including how to choose the alert payload fields your dedup key is built from, are in our guide to automated alert deduplication and grouping strategies.

Route to a person, not a room. An alert delivered to a shared channel belongs to everyone, which means it belongs to nobody, and the delay between arrival and ownership is dead time that shows up directly in your mean time to acknowledge. Attribute-based routing reads the service and environment off the payload and picks a responder. This is also the single biggest lever on mean time to resolution, which we cover at length in the guide to reducing MTTR.

Attach the runbook to the rule. A page that arrives with the procedure attached converts to action faster, and more importantly it makes the next actionability review easy: if nobody has ever opened the runbook, you have learned something about the rule.

Step 8. Put renewal on the calendar, not review

The difference matters. A review is an open-ended obligation that needs a champion and dies when that person changes team. A renewal is a dated event with a defined output: the rules expiring this month either get renewed or they lapse.

Monthly is enough for most teams. Thirty minutes. Walk the expiring set and the top ten noisiest rules from step 1, and ask the step 2 question again. Two outcomes are possible and both are fine.

Then recalculate the four numbers. If actionability is not climbing, the deletes were too timid, which is the normal failure and is easily fixed by running step 3 again with the delete bucket target enforced.

Where tooling helps

Tooling cannot decide which alerts should exist, and no vendor can give your team permission to remove one. What it can do is make the surviving alerts cheaper to handle and make the measurement above possible at all.

What is worth automating

• Deduplication and grouping before anything reaches a human. One failure, one incident, regardless of how many rules noticed it.
• Attribute-based routing to a named responder. The payload already contains the service, environment and severity. Something should read those and pick a person.
• Escalation with timeouts. If the primary has not acknowledged within a set window, it moves to the secondary automatically, without anyone noticing that it did not get picked up.
• Alert history you can actually query. You cannot calculate an actionability rate or a rule concentration if your alerting tool cannot tell you what fired and what happened next. This is the least glamorous item on the list and the one that blocks everything else.
• Snoozing at the integration level. For the misbehaving-source case in step 4, so the fix does not have to be deleting an integration you need.

What tooling will not fix

• The permission problem. No product ships a feature that makes your organisation comfortable deleting an alert rule. That is steps 4 and 5, and they are written on a page, not configured.
• An undefined severity scale. Automating a decision nobody has made produces faster confusion.
• Alerting on the wrong things. If your monitoring watches CPU graphs rather than user-facing outcomes, routing that perfectly just delivers the wrong signal to the right person.

If you are evaluating tools against this list, the useful question is not how many integrations it has. It is whether it can deduplicate and group on fields you choose, route on the contents of a payload, escalate on a timeout you control, and export enough history to calculate the four numbers above.

For what it is worth, that is the part of the problem All Quiet is built for, with the Standard plan published at $4.99 per user per month and no sales call required to see it. It will not tell you which alerts to delete, and it cannot make that decision safe inside your organisation. If your ten worst rules produce 70% of your volume, the honest advice is to go and delete those ten rules first, with whatever you are already paying for. Come back for the compression layer when the stock is under control.

The trade-offs nobody mentions

You will eventually miss something, and you should decide now whose fault that is. This is the real cost and it should not be minimised. A smaller alert set has less coverage, and at some point a removed rule would have caught something. The argument for doing it anyway is that an ignored alert provides no coverage either, while costing a real person real sleep. But that is a trade, not a free win, and step 4 point three exists precisely because this will happen.

Actionability rate is gameable and a bad target. Tie it to anyone's performance and you will get the number without the improvement, most likely by reclassifying trivial acknowledgements as actions. Track it as a diagnostic, look at it next to pages per responder, and never put it in a review.

Expiry dates create a renewal treadmill if you set the window too short. Twelve months is deliberate. At three months you have created a recurring meeting that people learn to rubber-stamp, which is worse than not having it, because now the ritual provides false assurance.

A very low alert count can hide a monitoring gap. A team with 12 alerts a week and a 95% actionability rate looks excellent and may simply not be watching much. Read alert volume next to incident volume and customer-reported issue volume. If customers are finding your outages before your alerting does, you have the opposite problem and this entire guide is the wrong one.

This framework assumes you have enough incidents to learn from. Below roughly one significant incident a month, the actionability data is too sparse to be meaningful and you should keep broader coverage deliberately. Run steps 1, 2 and 5, skip the aggressive deleting, and revisit when you have volume.

A 30-day version

If you want concrete work rather than a philosophy:

• Week 1. Export 90 days of alert history. Calculate the four numbers. Change nothing else. If rule concentration is above 60%, you are nearly done already.
• Week 2. Write and circulate the three sentences from step 4. Get agreement on the third one specifically, in writing, from whoever runs postmortems.
• Week 3. Run steps 1 to 3 on your top 20 noisiest rules only. Delete the delete bucket. Demote the demote bucket. Twenty rules is enough to move the numbers and small enough to finish.
• Week 4. Add owners and twelve-month expiry dates to everything that survived, and put the first renewal in the calendar. Recalculate the four numbers and publish the before and after.

At the end you will know your actionability rate, which is more than most teams reporting an alert volume figure can say, and you will have a mechanism that keeps working without anyone championing it.

Frequently Asked Questions

Who should decide which alerts get deleted?

The team that owns the service the alert watches, recorded as a team decision rather than an individual one. The single most effective change is giving each rule an owning team and a twelve-month expiry date, so rules lapse unless someone renews them. That moves the burden of justification from the person who wants a rule removed to the person who wants it kept, which means nobody has to take personal risk to reduce alert volume.

What if we delete an alert and then miss a real incident?

Assume this will happen occasionally and decide in advance that it is an expected outcome of the process rather than an individual's mistake. Keep removed rule definitions in version control so restoring one is a quick revert, record removals under the team's name, and get written agreement before you start that a missed incident from a removed rule is reviewed as a process outcome. If you cannot get that agreement, do not start deleting, because the process will not survive its first bad outcome.

What is a good actionability rate for alerts?

Above 30% is a reasonable working target, meaning at least three in ten alerts lead to someone doing something they would not otherwise have done. Below 20% the alerting is training responders to ignore it. Teams measuring it for the first time commonly land between 5 and 15%. Treat it as a diagnostic you track over time rather than a target anyone is measured against, because it is easy to game by reclassifying trivial acknowledgements as actions.

Does AI-assisted triage fix alert fatigue?

It helps with correlation, enrichment and summarisation, so responders start from context rather than a raw payload, but it does not decide whether a rule should exist, because that requires knowing what your organisation considers an acceptable failure. There is also a risk worth naming: making noise cheaper to tolerate removes the pressure that would eventually force someone to remove the rules generating it, so the underlying volume keeps growing under a layer that makes it feel manageable.