old stuff

2025-SELF/presentation.md

@@ -7,7 +7,7 @@ patat:
         left: auto
         right: auto
         top: auto
-    incrementalLists: true
+    incrementalLists: false
     speakerNotes:
         file: ./notes.txt
     pandocExtension:
@@ -15,35 +15,26 @@ patat:
         - emoji
     links:
         osc8: true
+    images:
+        backend: w3m
+        path: '/usr/libexec/w3m/w3mimgdisplay'
 ...
 
-<!--
-So for the past few years I've been coming to SELF, and hearing all these great
-talks, and having just got my amateur radio license at last year's conference,
-I had an idea for something a bit chaotic.
-
-I had some really grand ideas for this, and had to cut it back a bit due to
-time constraints with releases of Rocky, but I still think we're going to have
-a good time, and perhaps there's some information that can be gleaned at the
-end.
-
--->
-
 A Rocky Linux Disasterpiece
 
 # Background
 
+## Who Am I
 <!--
-So for the past few years I've been coming to SELF, and hearing all these great
-talks, and having just got my amateur radio license at last year's conference,
-I had an idea for something a bit chaotic.
-
-I had some really grand ideas for this, and had to cut it back a bit due to
-time constraints with releases of Rocky, but I still think we're going to have
-a good time, and perhaps there's some information that can be gleaned at the
-end.
 
 -->
+
+* Neil Hanlon
+* Open Source Systems Engineer @ CIQ
+* Founder @ Rocky Linux
+* Amateur Radio: N1HAN
+* Complaints (mine): bsky.app/profile/shrug.pw
+
 ## Why Are We Here
 
 <!-- 
@@ -82,77 +73,130 @@ As most solutions should, we began by taking an objective look at our needs and
     * Incident notifications
     * Monitoring visibility
     * Escalations that aren’t spam
+    * A bit of levity
+
+## Positively Ludicrous Notifications
+
+![](insane.jpg)
 
 # tHe FuTuRE
 
 ## Our Replacement Stack
+<!--
 
-* Prometheus ➔ Alertmanager
+-->
+
+* Prometheus (Mirmir) ➔ Alertmanager
 * Alertmanager ➔ Iris.claims
 * Iris.claims ➔ Oncall.tools
 * Notifications ➔ Fan-out to:
     * Email
     * SMS/Voice
     * MQTT ("the party bus")
-    * Restaurant pagers
-    * Meshtastic
-
-. . .
+        * Restaurant pagers
+        * Meshtastic
 
 **Ops Tools**
 
-* Grafana
-* OpenSearch
 * Ansible
+* Grafana
+* Loki
+* OpenSearch
+
+
+## Prometheus / Alertmanager
+<!--
+-->
+
+### Prometheus / Mirmir
+
+* Time-series database / monitoring system 
+* Scrapes metrics from services, allows querying via PromQL
+* Infinitely extensible
+* Mirmir can be added as you scale
+
+### Alertmanager
+
+* Handles alerts generated by Prometheus (and other sources)
+* Groups, Inhibits, Silences, and Dedupes
+* Direct routing to notification channels based on rules
+* Centralized alert delivery / incident management
 
  
+## Iris.claims / Oncall.tools
+
+<!--
+Both from linkedin
+Enriches with routing rules, escalations, schedules
+-->
+
+### Iris.claims
+
+* Incident notification and management system 
+* Enriches Alertmanager alerts
+* Handles notification to owners and stakeholders
+* Bridge between alerting tools and human scheduling
+* Supports prometheus metrics for monitoring
+    * who does watch the watchmen?
+
+### Oncall.tools
+
+* Lightweight on-call scheduling
+* Calendar-based scheduling w/ an API
+* Hydrates Iris with up-to-date schedule information
+
+## iris.claims
+![](iris.png)
+## oncall.tools
+![](oncall.png)
+
+# Inputs and Outputs
 ## Monitoring Inputs
 
+<!--
+
+-->
+
 * Prometheus/OpenTelemetry exporters
 * UptimeKuma for endpoint monitoring
     * TLS cert expiration checks
 * Disk usage, load average, etc.
 * Service-specific KPIs
 * Cronjob-driven questionable life choices
+    * "The compose failed"
+    * "Doctors hate this one weird trick for cleaning /tmp"
+
 
 ## Notification Channels
+<!--
 
+Iris supports some channels out of the box, like Slack, Email, and Voice/SMS.
+
+I've been working on adding generic support for MQTT, to integrate with basically anything.
+-->
 
 * Email: would be better with stamps
 * Voice/SMS: Twilio, CallMeBot
     * we don't, but you could
 * ~~Slack~~ Mattermost (Webhooks)
-* MQTT: The glue for all things absurd:
+* MQTT: The glue for all things absurd
     * Restaurant pagers from eBay
     * Meshtastic LoRa nodes
-    * (Almost) Radiograms and APRS
-
-## Live Demo Preview
-
-* Trigger alert in Grafana
-* Alertmanager routes
-* Iris creates incident
-* Oncall.tools decides who to ping
-* All aboard the party bus
-    * Pager buzzes
-    * Meshtastic lights up
-    * Maybe even DMR SMS
+    * POCSAG Pagers
+    * A bell attached to a wire controlled by a servo
+    * Radiograms and DMR SMS
 
 ## MQTT: The Real Party Bus
 
-
 <!--
 
-
-
-
 really your imagination is the limit..
 -->
 
 * Decouples everything
 * pub/sub means all alert channels are equal
 * Not tied to a single vendor's services
-* Simple to glue to:
+* Simple to fan out to:
     * Restaurant pagers
     * Meshtastic / LoRa endpoints
     * Bash
@@ -160,36 +204,157 @@ really your imagination is the limit..
     * Home Assistant?
     * Jira??
 
+## ~~Live Demo Preview~~ How it should've worked
+<!--
 
+-->
+
+* Trigger alert in Grafana
+* Alertmanager routes
+* Iris creates incident
+* Oncall.tools decides who to ping
+* All aboard the party bus
+    * Pagers buzz
+    * Meshtastic devices go beep beep
+    * Hopefully someone wakes up
+
+
+# Meshtastic
+
+## Meshtastic
+
+* Open-source, off-grid, mesh communication using LoRa (LongRange) radios
+* Low power, long distance comms w/o cellular or network access
+* *Can* be used in conjunction with Internet via MQTT
+* End-to-end encryption
+    * Nodes can still forward packets even if they cannot decrypt them
+* Originating a message from MQTT to be sent is possible, if difficult
+
+## MQTT Integration
+
+* Private MQTT broker must be configured on meshtastic device 
+    * okToMqtt, server settings
+    * JSON output
+* **Including** adding a channel named `mqtt` and enabling Downlink on it
+
+```python
+client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
+client.connect('localhost')
+client.publish("msh/US/2/json/MQTT", {
+    "from": "860292936",
+    "channel": 1,
+    "type": "sendtext"
+    "payload": "Server's on fire, yo."
+})
+```
+
+Future: Listen for and process acknowledgements from oncall
+
+# Restaurant Pagers
+
+## Retekess TD174
+
+* Found on ebay for $30
+* Missing one pager, also one broken
+* Not as plug-and-play as I hoped with a Flipper
+* Hardware mod on the base station is probably a better play
+
+```
+Filetype: Flipper SubGhz Key File
+Version: 1
+Frequency: 433889000
+Preset: FuriHalSubGhzPresetOok650Async
+Protocol: SMC5326
+Bit: 25
+Key: 00 00 00 00 01 06 B0 40
+TE: 326
+```
+
+# POCSAG and more
+## POCSAG
+
+* Struggled finding a pager in time for a demo
+* Many hotspots support transmitting pages
+* hampager.de
+
+## Radiograms
+
+```
+52 R HXG N1HAN 11 CHARLOTTE NC JUN 15 
+SOME STAKEHOLDER KC1ABC
+SOMEPLACE SOMEWHERE SOMEZIP
+BT
+YOUR TLS CERTIFICATE FOR ROCKYLINUX
+DOT ORG EXPIRES NEXT MONTH
+73
+BT
+NEIL N1HAN
+AR
+```
+
+Maybe don't do this? I dunno
+
+## Possibly useful
+
+### ntfy.sh
+
+* Simple HTTP-based pub/sub push notification service
+* Push notifications w/ actions
+* Can deploy yourself or use hosted
+    * iOS/Android apps
+
+### DMR SMS
+
+* Send an SMS to your digital-mode radio!
+
+# On-Call At-Home 
 ## On-Call At-Home 
 
+<!--
+
+-->
+
 * Leak in the basement!
     * HA sends MQTT alert
     * Buzzer sounds
     * Lights flash
     * Pager buzzes (for real)
+    * You go turn off the water
 * Easily reuses same infra stack
 
 
 ## What Didn't Make It
 
+<!--
+
+-->
+
+* My demos
 * Radiograms
 * Winlink->MQTT pipeline
 * APRS group messaging (CQSRVR)
+* DMR SMS
 * Whisper network / pigeon relay
+* The servo-controlled bell thing
 
 ## What We Learned
 
+<!--
+
+-->
+
 * You *can* replace SaaS alerting
 * MQTT is weirdly good at this
 * Hardware is fun again
-* LoRa is legit ops-grade
-* The pager lives
+* LoRa is approaching ops-grade, with limitations
+* The pager lives!
 
-## Q&A + Live Triggers
+## Q&A
+
+<!--
+
+-->
 
-* Trigger alert with curl or form
-* Watch things blink and buzz
 * Ask away
 
 ## Thanks!