More regular reminders for issues that need triaging

[armenzg]

Requirements

1. Buckets:
   1. overdue = overdue
   2. act fast = less than 4 hours to deadline (or deadline is after end of day)
   3. triage queue = remaining tickets
2. Check queue once per hour.
   1. This is in addition to the immediate notification when a new ticket comes in.
3. If any are overdue or in act fast, post.
4. If we haven't posted in four hours, post.
5. Message format:

Hey! You have some tickets to triage:

🚨 Overdue 😰

• "[$github_issue_title]" 8 hours overdue
• "[$github_issue_title]" 2 hours overdue
• "[$github_issue_title]" 1 hour overdue

⌛️ Act fast! 😨

• "[$github_issue_title]" 15 minutes left

⏳ Triage Queue 😯

• "[$github_issue_title]" 8 hours left
• "[$github_issue_title]" 24 hours left
• "[$github_issue_title]" 22 hours left
• "[$github_issue_title]" 36 hours left
• "[$github_issue_title]" 48 hours left 👈 quick win?

Original

The workflow team receives Issue pending alerts in Slack, however, if an alert is missed we will not hear about it anymore.

For instance, this issue was not triaged and I got pinged on Slack yesterday (9 days after the issue was tagged for triaged)

We should get reminded every day until we triage the queue.

Thanks!

To Do

• backfill offices for channels with existing triaging rules
• Rework scheduler triage notifications eng-pipes#376
• https://github.com/getsentry/ops/pull/5857
• Add hourly notification support eng-pipes#377
• https://github.com/getsentry/ops/pull/5863
• Send slack message to OSPO channel if a channel is not subscribed to any offices eng-pipes#381

[chadwhitacre]

Thanks @armenzg, I thought we had something for this (at least for issues over triage) but I can't find it now. I agree this would be better than current process which involves human review on a weekly cycle, expensive and slow.

[BYK]

@chadwhitacre the scheduled job trigger is GCP

[BYK]

It is pinging an open endpoint on eng-pipes so you can probably change the logic there.

[chadwhitacre]

the scheduled job trigger is GCP

Oooooooh! Thanks @BYK! 😅