Search

Alert Unification

Introduction

Bettrdata has a number of slack alerts to let the ops and other teams know when there is an issue. However, the number of channels is large and makes it difficult to track real problems from noise.

Alert List

The slacks now included

  • eng-system-event-notice
    • Used to tell whether bettrdata instances are being shut down or brought up. May include comments
  • ops-accuzip-event
    • Alerts when data is being passed to accuzip for NCAO processing
  • ops-net60
    • A special channel for zendesk tickets specific to net60
  • ops-notifications
    • Zendesk ticket alerts
  • ops-prismatic-alerts-channel
    • Alerts specific to prismatic issues
  • ops-support-checkmk
    • CheckMK alserts
  • ops-zendesk-status
    • Shows list of zendesk tickets by priority and assigned state

Unification of Channels and Obstacles

Bringing a number of the channels together would help in making the job of covering issue more straightforward and smoother. These channels can be unified into new unified channels:

  • ops-info
  • ops-alerts

Challenges to Better Slack Alerts

However some channel messages may need to be adjusted in order to minimize noise. There also may need to be some division of alert types that are presently going into the same channel. For instance, accuzip issues could go to one channel and accuzip successful processing could go to another channel.

Ops-Info Channel

This channel will help in informational messages that do not necessarily need to be triaged at the moment but kept in view. Channels to include:

  • eng-system-event-notice
  • ops-accuzip-event
    • Maybe this could be bucketed
    • Could real alerts be diverted to the alerts channel?
  • ops-net60
    • This should be folded back into regular zendesk tickets
  • ops-notifications
    • Could we divide by task, rather than issue or problem
  • ops-zendesk-status

Ops-Alerts Channel

This channel would be for message telling clearly alert about something that needs to be dealt with

  • ops-prismatic-alerts-channel
  • ops-support-checkmk
    • This channel will need to be tuned quite a bit before adding
  • ops-notifications
    • For issue or problem tickets
  • ops-accuzip-event
    • For issues