Skip to content

Notifications

Two-channel notification system for cron jobs across the stack: Telegram (substantive events, fast) + Resend email (always, paper trail). UI at /notifications for control + history. Built 2026-05-03.

Channels

Telegram

  • Bot: @otq_moonbot (chat_id 8196786680)
  • Token in ~/.zshrc as TELEGRAM_BOT_TOKEN
  • Used for: cron success/first-failure, OTQCheckinAgent check-ins, morning summaries, replies to corrections/questions, ad-hoc alerts
  • Push delivery, ~instant, mobile-friendly
  • Same bot used by the OTQCheckinAgent Worker via webhook

Resend

  • Domain: updates.caseymanos.com (DNS in Cloudflare, all records grey-cloud / DNS-only)
  • API key in ~/.zshrc as RESEND_API_KEY
  • Used for: every cron run (success or fail) for paper-trail email
  • Per-source From addresses for Gmail filter routing: | Source | From | |---|---| | podcast-sync | podcast@updates.caseymanos.com | | warehouse-sync | garmin@updates.caseymanos.com | | monitor-poll, verify-studies, hot-briefing | research@updates.caseymanos.com | | test, unknown | pipeline@updates.caseymanos.com |

Mapped in data-ingestion/notify.py:SOURCE_SENDERS.

Sources

Source Cadence Trigger TG Email
podcast-sync Sun 6am PT (launchd) ~/data-ingestion/scripts/podcast-sync.sh yes (every run) yes
warehouse-sync Daily 7am PT (launchd) ~/garmin-warehouse/scripts/daily_sync.sh success or first-fail only yes (every run, with per-pattern runbook on fail)
monitor-poll manual / UI button uv run python monitor_setup.py poll yes (substantive only) yes
verify-studies manual verify_studies flow only on new contradictions yes
hot-briefing manual hot_briefing.py yes (on completion) yes
otq-checkin (Worker) 8pm PDT (runCheckin) + 8:30am PDT (runMorningSummary) Cloudflare Worker cron every fire no — Worker doesn't email

UI

/notifications at https://warehouse.caseymanos.com/notifications (uvicorn on :8765). Built with FastAPI + Jinja2 + Tailwind + HTMX.

Sections: - Channel cards — Telegram + Resend with "test send" buttons (toast confirmation) - Per-source toggles — enable/disable TG and email per source independently - Schedule + invocation table — what fires when, with "run now" buttons - Live deliveries log — auto-polls every 5s, shows recent attempts

Storage

File Contents
~/.config/notify-settings.json Per-channel + per-source toggle state
~/garmin-warehouse/logs/notifications.jsonl Delivery log: every send attempt with result, recipient, source, ts

Gmail filter strategy

Export at ~/data-ingestion/scripts/gmail-filters.xml. 5 filters:

  1. from:podcast@updates.caseymanos.com → label Pipelines/Podcast
  2. from:garmin@updates.caseymanos.com → label Pipelines/Garmin
  3. from:research@updates.caseymanos.com → label Pipelines/Research
  4. from:pipeline@updates.caseymanos.com → label Pipelines/Test
  5. Failure-overlay: subject contains "failure" → keep in inbox + star (overrides the per-source labels so failures stay visible)

To re-import after a Gmail account migration: Gmail Settings → Filters and Blocked Addresses → Import filters → upload the XML.

Why this exists

Cron jobs were silent before 2026-05-03. Failures (especially expired Garmin garth sessions) sat for days unseen. Now:

  • TG for substantive events that need a response
  • Email for paper trail (so I can grep "what happened on YYYY-MM-DD")
  • UI for control + history (toggle a source off when traveling, etc.)
  • Failure-overlay Gmail filter so a failure email beats other rules

Failure runbook block

daily_sync.sh builds a per-failure-pattern runbook into the email body. Match patterns include:

  • *garmindb*garth*session*expired* → Garmin SSO recovery steps
  • *refresh-icu* → intervals.icu API key recovery
  • *kb/sync* → Voyage embedding troubleshooting

See runbooks/daily-sync-failures.md for the full table.