All Quiet Masterclass: Stop Auto-Escalations for Minor Incidents

The Problem - Escalations for Minor Incidents Cause a Lot of Noise

If you’ve ever been on-call, you know how draining it is when alerts aren’t properly tuned to match the severity of incidents, leaving you overwhelmed by excessive noise. The critical issues? They matter. The rest? Well, at least no reason to get up am 2 a.m.

In this article, we’ll share a practical tweak to stop your team's escalations for certain types of incidents. Because incident management should be about resolving what matters - not drowning in alerts that don’t.

The Solution - Create a Rule to Stop Escalations

I find that the best way to understand a solution is through a practical example that shows exactly how it works. Hopefully, you’ll agree — and the example below will make it easy for you to set everything up. Still have questions? No problem — shoot me a message anytime.

First - Understand Your Team's Escalation Policies

To kick off our example, we need to look at our example team's schedules and escalations:

• For simplicity, each escalation tier has only one on-call member that is online 24/7.
• If noone reacts in tier 1, incidents are escalated to tier 2 after 5 minutes.
• The same applies for escalation from tier 2 to tier 3.

Picture this: a minor incident pops up. Tier 1 is busy with something else and doesn’t react immediately because, well, it doesn’t seem urgent. Five minutes later, the incident escalates to tier 2. Same story — no immediate action. Then it escalates again, this time to tier 3. Suddenly, what started as a trivial issue has disrupted three people, pulling their focus away from more important work.

This is why we need to tackle unnecessary escalations head-on. By mitigating escalations for low-priority incidents, we can keep the team focused on what truly matters.

Second - Set Up the Escalation Ending Rule

If you want your tier 1 members to be informed about minor incidents but don’t want to escalate to others unnecessarily, you can set up a straightforward routing rule. It’s an easy way to keep things under control without disrupting the rest of your team.

• To create the rule, go to "Routing & Workflows" in your Web App.
• Create a rule and give it a name. If your team already uses routing rules, we recommend adding the new rule to your existing list. This ensures that all rules are executed in the correct order.
• Start setting up the rule by defining the conditions. First, we want to make sure it is applied for Minor (small) Incidents only.
• Moreover, we need to add that the action is only triggered for the incident intent "Created". Why? If the current intent is "Created", this means that after incident creation nobody has acknowledged the incident and that auto-escalations are still active.
• Now, we need to add the action. First and foremost, we want to trigger it with a certain delay to the conditions. Here, we decide for 2 minutes. To stop auto-escalations, the delay could also be 1, 3 or 4 minutes. The only thing that matters is that the action is delayed and that the delay is below 5min, as our auto-escalation to tier 2 happens 5 minutes after incident creation in this example (see escalations section above). Also, a delay is necessary since no delay for the action would prevent the tier 1 members to be informed about incident creation. Instead, they would only be informed about the action triggered by this rule.
• The action: We suggest to mark the incident as "investigated". This will immediately mark the incident as acknowledged and stop auto-escalations. Note: The action will only be applied if noone already reacted to the incident within the 2 minutes (the delay we added after the incident matches the conditions) before.

The corresponding code snippet for our Terraform Provider (version 2.2.1). Please adjust the ressource names to you config accordingly:

  
    resource "allquiet_routing" "dont_escalate_minor" {
      team_id      = allquiet_team.my_team.id
      display_name = "Don't Escalate Minor Incidents"
      rules = [
        {
          conditions = {
            severities = ["Minor"]
            intents    = ["Created"]
          },
          actions = {
            add_interaction          = "Investigated"
            delay_actions_in_minutes = 2
          }
        },
      ]
    }
  

Third - The Streamlined Incident Response After Applying the Rule

I created an example incident with "minor" severity to demonstrate the functioning of the rule.

• As you can see, if nobody reacted the incident after 2 minutes, the routing rule automatically marks the incident as "investigated".
• Remember, without interaction the incident would have been auto-escalated after 5 minutes. However, I resolved the incident 7 minutes after the action was triggered via the rule. Without any auto-escalation. Without unnecessary noise.

Final Thoughts

Reducing noise in incident management helps building a more resilient, focused system. By applying this simple hack to your escalation function, you can prevent unnecessary distractions, keep your team aligned, and ensure their attention is reserved for the incidents that truly matter.

Remember, incident management isn’t a one-size-fits-all approach. Every team has its own unique challenges, and our routing rules & workflows give you the flexibility to adapt your workflows accordingly. Stopping minor incidents from auto-escalating is only one of myriads ways to adapt All Quiet to your needs.

So, take a moment to review your current escalation setup. Is it working for you, or is it creating unnecessary churn? With the strategies outlined here, you can take control, reduce the noise, and build a more efficient on-call process that works for your team — not against it.

Have questions or thoughts? Or maybe a totally different issue and want to know how All Quiet can solve it? Reach out anytime!

Peer
CEO & Co-Founder of All Quiet