A better way to manage on-call alerts.

Alerting is a tough thing to balance.

• You either get too many alerts or too little.
• How do you figure out if an alarm is an actual incident?
• How do you make sure that the relevant teams are looking at the issue?

This is not an attempt to answer any of those questions.

This is me trying to find some kind of use out of a 50$impulse Ebay buy… Please do not do this in$production ;)

I bought an Okidata Microline 184 TURBO! 9 pin printer.

Just look at that amazing turbo logo. You know it goes fast by just looking at it.

### Why?

I’m not 100% sure but I feel like part of my brain was hardwired to associate the sound of dot matrix printing to reliability and comfort. But I’m too young to have even owned one so I have no idea where that fascination comes from.

Just listen to that amazing purr (not my exact model but the sound is close enough).

https://youtu.be/A_vXA058EDY?t=33

### Finding something to do with it.

So after a delivery (that required some explaining to the significant other), I figured I might as well find a use for it.

At \$DAYJOB, we process alarms. A lot of alarms. Some alarms are good, some alarms are bad. Some are noise, some are significant. Part of the work of a DEVOPs team is to drive the vision that alarms should mean something and that too many alarms is as worse as too little.

But, when your logic is good, you want to make sure that critical alarms are properly highlighted.

### Now, this is podracing (printing)!

This is where I thought that I could maybe get the printer doing something useful.

I run a small homelab of a couple of Dell R610 / R720 (blog post on those coming later). These are running Proxmox, ESXI and Kubernetes clusters. I have some alerting setup with the Prometheus stack and I figured I could give myself some visibility on the alerts! Sure, I could use Slack, SMS or Email. But printing is where it’s at. It’s reliable and makes enough noise to wake you up at 3 in the morning. Take that Pager Duty!

#### Overview of the flow

graph LR; A[Prometheus] --> B[Alertmanager] -->|WEBHOOK| C[Laptop FastAPI] -->|PRINT| D[Okidata 184 Turbo] linkStyle default stroke-width:2px,fill:green,stroke:blue;
• Prometheus has a rule configured for X state.
• State is triggered