Plan monitoring and observability for course management platform

[alexeygrigorev]

Goal

Add practical monitoring and observability for the course management platform without self-hosting Grafana, prioritizing free or low-cost hosted options.

We want to know what is happening around releases and catch application health problems early:

• unhandled errors and exceptions
• release/deploy health and regressions
• important student/staff events
• page/API availability
• submission/scoring/email pipeline health

Current app inventory

The app is a Django project deployed to AWS ECS from GitHub Actions:

• GitHub Actions builds Docker images, pushes to ECR, and updates ECS task definitions in .github/workflows/deploy-dev.yaml, .github/workflows/deploy-prod.yaml, and deploy/deploy_*.sh.
• deploy/update_task_def.py writes the Docker tag into the ECS VERSION environment variable.
• /api/health/ returns {"status": "ok", "version": settings.VERSION} from api/views/health.py.
• Django already emits JSON logs to stdout in non-local environments through course_management/settings.py.
• No Sentry, OpenTelemetry, AWS X-Ray, Datadog, New Relic, Honeybadger, or hosted Grafana dependency was found in pyproject.toml.
• Datamailer has useful domain-level health data already stored in data.models: DatamailerOutboxEvent, DatamailerOutboxDispatchRun, DatamailerContactEvent, and DatamailerSendAudit.
• There are management commands that already summarize Datamailer health: datamailer_outbox_status, datamailer_send_status, and datamailer_callback_status.

Important events to monitor

Authentication and identity

• Login page views: accounts/views/login.py.
• Successful/failed OAuth login and account linking: accounts/auth.py via ConsolidatingSocialAccountAdapter.pre_social_login.
• New users created from social login: ConsolidatingSocialAccountAdapter._create_user_for_email.
• Multiple existing users found for the same social email: currently logged as warning in accounts/auth.py.
• Token-auth API failures: accounts/auth.py::token_required.
• Staff impersonation start/stop: django-loginas under /admin/ and accounts/views/impersonation.py::stop_impersonating.

Privacy note: current OAuth logging includes full provider response JSON. Before expanding log retention, review this for PII/token exposure and either remove it or redact it.

Page/API traffic and availability

• Public course list/course/detail pages from courses/urls.py.
• Student pages: course dashboard, homework, project, leaderboard, enrollment.
• Staff admin pages under cadmin/urls.py.
• API requests under api/urls.py, especially staff-token mutation endpoints.
• /api/health/ uptime and version checks.
• 4xx/5xx counts, request latency, and response sizes from ALB/ECS/gunicorn access logs if available.

Student lifecycle events

• Registration campaign submitted: courses/views/registration.py::registration_campaign_view.
• Enrollment created/updated/toggles changed: courses/views/course_enrollment.py.
• Account preference/toggle/timezone/email-preference changes in accounts/views/*.
• Leaderboard complaint submitted/resolved: courses.models.course.LeaderboardComplaint and cadmin/views/enrollment.py.

Homework events

• Homework viewed and submitted: courses/views/homework.py and courses/views/homework_submission.py.
• Submission update versus first submission: homework_submission_for_user.
• Homework validation errors: courses/views/homework_post_preview.py.
• Homework scoring started/completed/failed: courses/scoring.py and API/staff routes.
• Staff edits to submissions and correct answers: cadmin/views/homework.py, cadmin/views/homework_submission_edit.py.
• Score notification sends: Datamailer outbox/send audit events.

Suggested event fields: event_name, course_slug, homework_slug, homework_id, submission_id, user_id, enrollment_id, is_update, question_count, duration_ms, status, error_code.

Do not log answer text or free-form comments.

Project and peer review events

• Project viewed/submitted/deleted: courses/views/project.py and courses/views/project_submission_edit.py.
• Project validation errors.
• Peer reviews assigned: courses/project_assignment.py.
• Peer review submitted: courses/views/project_eval_submit.py and courses/views/project_eval_submit_save.py.
• Optional peer review add/delete: courses/views/project_eval_actions.py.
• Project votes: courses/views/project_eval_submit.py::project_eval_vote_response.
• Project scoring started/completed/failed: courses/project_scoring.py and API/staff routes.
• Staff submission edits: cadmin/views/project_submission_edit.py.

Suggested event fields: event_name, course_slug, project_slug, project_id, submission_id, review_id, reviewer_enrollment_id, user_id, assigned_count, submitted_count, passed_count, duration_ms, status, error_code.

Do not log repository contents, notes to peer, or free-form comments. GitHub repo URLs and commit IDs are probably acceptable as structured fields, but avoid over-indexing them in metric dimensions.

Staff/API mutations

• Course/homework/project create/update/delete through api/views/*.
• Homework question create/update/delete through api/views/questions.py.
• Registration campaign create/update through api/views/registration_campaigns.py.
• Certificate/graduates imports and bulk certificate update endpoints.
• Staff-triggered homework/project scoring and review assignment.
• Unauthorized/invalid token attempts.

Suggested event fields: event_name, actor_user_id, course_slug, object_type, object_id, method, path_name, status, error_code.

Datamailer and background-like work

• Outbox enqueue/dispatch/ack/retry/fail/dead from course_management/datamailer_outbox*.py.
• Outbox run success/failure from DatamailerOutboxDispatchRun.
• Send audit success/failure from DatamailerSendAudit.
• Webhook callback accepted/duplicate/rejected from api/views/webhooks.py and validation helpers.
• Registration confirmation, homework/project confirmation, score notifications, certificates, peer-review notifications.
• Scheduled process_datamailer_outbox execution should have heartbeat/check-in monitoring.

Recommended monitoring architecture

Phase 1: cheapest useful baseline

Use CloudWatch for infra/log retention plus Sentry or Honeybadger for application errors and release visibility.

1. Keep structured JSON stdout logs and ship ECS task logs to CloudWatch Logs.
2. Add log fields consistently: event, release, environment, request_id, user_id, course_slug, object_type, object_id, status.
3. Add Sentry SDK or Honeybadger for Django exceptions.
4. Set release=settings.VERSION and environment=dev|prod, matching the ECS VERSION value already set by deployment.
5. Add GitHub Actions release/deploy notification to the error tracker after deploy.
6. Add an external uptime monitor for /api/health/ and one or two key public pages.
7. Add CloudWatch alarms for ECS service health, 5xx rate, task restarts, memory/CPU, and log error patterns.
8. Add a daily/hourly scheduled health script that runs existing Datamailer status commands and emits one compact JSON log line or CloudWatch metric.

This gives immediate error tracking and release correlation without running Grafana.

Phase 2: app-level events

Add a tiny internal event helper, for example monitoring/events.py, that logs a structured event and optionally increments metrics later.

Initial events:

• auth.login_success, auth.login_failed, auth.user_created, auth.social_account_linked
• registration.submitted
• enrollment.created, enrollment.updated
• homework.submitted, homework.validation_failed, homework.scored
• project.submitted, project.deleted, project.peer_reviews_assigned, project.review_submitted, project.scored
• api.mutation, api.auth_failed
• datamailer.outbox_failed, datamailer.outbox_backlog, datamailer.send_failed, datamailer.callback_received

Important: do not emit high-cardinality dimensions as CloudWatch custom metrics. Keep detailed IDs in logs/error context. Use metrics only for low-cardinality counters such as event name, status, environment, and object type.

Phase 3: dashboards and alerts

Build a small number of hosted dashboards/alerts:

• Release overview: deploy version, error count since deploy, 5xx count, health check status.
• Student activity: registrations, enrollments, homework submissions, project submissions, peer reviews.
• Course operations: scoring runs, review assignment, leaderboard updates.
• Datamailer: outbox backlog, failed events, last successful run, send audit failures, callback freshness.
• API/admin: mutation counts, unauthorized requests, staff-triggered destructive actions.

Option matrix

Option	Covers	Likely low-volume cost	Pros	Cons
AWS CloudWatch Logs + alarms	ECS logs, error log patterns, 5xx metrics, CPU/memory, dashboards	Often near free if logs stay under free/low GB volumes; official pricing includes paid ingestion/storage after free usage	Already AWS-native; no new vendor; easy ECS integration	Weak error grouping; custom metrics can get expensive with high-cardinality labels
AWS X-Ray / CloudWatch Application Signals	Traces, latency, service map	Cheap with sampling at low volume; AWS example shows 148,800 sampled traces costing $0.24 recorded after free tier	Native AWS tracing; good for slow requests and DB/API latency	More setup; trace volume needs sampling discipline
Sentry hosted	Django exceptions, release health, deploy correlation, traces, uptime/cron monitors, alerting	Free Developer plan for one user and small volume; Team starts at $26/mo annually	Best fit for “what broke after release?”; Django integration is straightforward; release/deploy support	Free plan is one user and limited events; not ideal as the only event analytics store
Honeybadger hosted	Exceptions, deploy tracking, logs/performance, uptime, check-ins	Free Developer plan for one user; paid Team starts around $26/mo	Simple all-in-one app health product; includes check-ins and uptime	Smaller ecosystem than Sentry; less common if future contributors expect Sentry
PostHog	Product/app events, page views, funnels, lightweight audit trail, optional session replay/error tracking	Free tier includes 1M analytics events and 100K exceptions/month	Best fit for “what are students/staff doing?” without turning CloudWatch into an analytics store; billing limits available	Not infra monitoring; must be careful with PII and free-form submission data
Better Stack	Uptime, logs/traces, incidents, status page, web events	Free tier includes small telemetry/monitoring allowances; responder/on-call is paid	Strong hosted uptime/logs/status-page workflow; no self-hosting	Broader platform; costs depend on telemetry volume and responders
Grafana Cloud hosted	Hosted Grafana/Loki/Tempo/Prometheus-style metrics/logs/traces	Free hosted tier exists; Pro starts at $19/mo plus usage	Satisfies “no self-hosted Grafana” while still using Grafana dashboards	More instrumentation/ops concepts than Sentry/Honeybadger; watch cardinality
UptimeRobot	External uptime checks	Free plan: 50 monitors with 5-minute interval; paid Solo starts at $7/mo annually	Very fast way to monitor /api/health/ and public pages	Uptime only; no app errors or release correlation
Healthchecks.io	Cron/job heartbeats	Free for 20 checks; paid Business is $20/mo or $16/mo annually	Very cheap Datamailer outbox/scheduled job heartbeat monitoring	Does not cover exceptions, product events, or request traffic
StatusCake	External uptime, SSL/domain/page-speed checks	Free plan has 10 uptime monitors at 5-minute intervals	Simple hosted uptime alternative to UptimeRobot	Less generous free monitor count than UptimeRobot
New Relic	All-in-one APM/logs/metrics/synthetics/errors	Free tier includes 100 GB/month data ingest and one full platform user	Large free ingest allowance and broad platform	Heavier than needed; costs can jump with additional full users or ingest
Honeycomb	OpenTelemetry traces/events and high-cardinality debugging	Free tier up to 20M events/month; Pro starts at $150/mo	Strong if tracing and high-cardinality backend debugging become central	Less direct than Sentry/Honeybadger for Django exception triage; paid tier is a bigger step up
EventBridge Scheduler + ECS task/management command	Scheduled health checks/Datamailer status emitters	EventBridge Scheduler has 14M free invocations/month	Cheap way to run status commands and heartbeat checks	Need to wire output into logs/metrics and alerts

Suggested decision

Recommended starting stack:

1. Sentry hosted for Django error tracking and release/deploy health.
2. PostHog for product/audit-style events such as login, registrations, page views, submissions, peer-review actions, and staff mutations.
3. CloudWatch Logs/alarms for ECS infra, log retention, and low-cardinality app counters.
4. UptimeRobot, StatusCake, or Sentry Uptime for /api/health/ and key page checks.
5. Healthchecks.io or Sentry Cron Monitoring for periodic datamailer_*_status and process_datamailer_outbox heartbeat/check-in monitoring.

Why this combination:

• It avoids self-hosting Grafana.
• It gives release-aware error tracking quickly.
• It reuses the existing health endpoint and VERSION deployment flow.
• It keeps CloudWatch custom metrics disciplined and cheap.
• It puts high-volume user/application events into a product-event tool with a generous free tier instead of into expensive high-cardinality CloudWatch custom metrics.

If the team wants the cheapest possible single external vendor, Honeybadger is a reasonable alternative to Sentry because it combines error tracking, deploy tracking, uptime, and check-ins. If product analytics becomes the primary need, PostHog should be the second integration after error tracking. If dashboards/log analytics become the primary pain, consider Better Stack or hosted Grafana Cloud next. New Relic has a generous free ingest tier but is probably more platform than this app needs initially.

Implementation checklist

• Add an error tracker SDK (sentry-sdk[django] or Honeybadger equivalent) with DSN/env vars disabled by default locally.
• Set release/version from settings.VERSION.
• Add GitHub Actions deploy/release notification for dev/prod.
• Add request ID middleware or configure one if already present.
• Add structured event helper for important domain events.
• Instrument homework/project submission, scoring, registration, enrollment, Datamailer, API auth failure, and staff mutation paths.
• Redact or remove OAuth provider response logging before retaining logs externally.
• Add CloudWatch log retention policy and alarms.
• Add external uptime check for /api/health/ and at least one public course page.
• Add scheduled Datamailer health/check-in job.
• Document event names and privacy rules.

Sources checked

• Sentry pricing and plan limits: https://sentry.io/pricing/
• Sentry Django integration: https://docs.sentry.io/platforms/python/integrations/django/
• Sentry releases/release health: https://docs.sentry.io/product/releases/
• Sentry GitHub Actions release automation: https://docs.sentry.io/product/releases/setup/release-automation/github-actions/
• AWS CloudWatch pricing: https://aws.amazon.com/cloudwatch/pricing/
• AWS EventBridge Scheduler pricing: https://aws.amazon.com/eventbridge/pricing/
• Honeybadger plans: https://www.honeybadger.io/plans/
• Better Stack pricing: https://betterstack.com/pricing
• Grafana Cloud free tier: https://grafana.com/products/cloud/free-tier/
• UptimeRobot pricing: https://uptimerobot.com/pricing/
• PostHog pricing and error tracking pricing: https://posthog.com/pricing and https://posthog.com/docs/error-tracking/pricing
• Healthchecks.io pricing: https://healthchecks.io/pricing/
• StatusCake pricing: https://www.statuscake.com/pricing/
• New Relic pricing: https://newrelic.com/pricing
• Honeycomb pricing: https://www.honeycomb.io/pricing

[alexeygrigorev]

Implementation update

Initial observability plumbing has been implemented behind a project-owned abstraction so app code is not coupled directly to a vendor SDK.

What changed

• Added course_management.observability with:
  • record_event(...) for product/audit/domain events.
  • report_exception(...) for future explicit exception reporting.
  • ping_check(...) for Healthchecks-style heartbeat pings.
• Added backend adapters:
  • noop: drops events, used by default during tests.
  • log: writes local/dev structured event logs.
  • posthog: sends event payloads to PostHog's capture API.
  • sentry: adds app events as Sentry breadcrumbs.
• Added Sentry initialization through settings, isolated in course_management.observability.sentry.
• Added monitoring_datamailer_health, a management command that emits compact Datamailer health and optionally pings Healthchecks.io.
• Added local tests for event schema, PostHog delivery payloads, Healthchecks pings, and the Datamailer health command.
• Added docs in docs/observability.md.

Events instrumented so far

• Auth/account:
  • auth.user_created
  • auth.login_success
  • auth.login_failed
  • auth.logout
  • auth.impersonation_started
  • auth.impersonation_stopped
  • account.toggle_updated
  • account.timezone_updated
  • account.email_preference_updated
• API auth:
  • api.auth_failed
• Registration/enrollment:
  • registration.submitted
  • enrollment.created
  • enrollment.updated
  • enrollment.toggle_updated
• Homework:
  • homework.submitted
  • homework.submission_rejected
  • homework.validation_failed
  • homework.scoring_started
  • homework.scoring_failed
  • homework.scored
• Projects/reviews:
  • project.submitted
  • project.deleted
  • project.validation_failed
  • project.vote_updated
  • project.review_unauthorized
  • project.review_submitted
  • project.optional_review_added
  • project.optional_review_deleted
  • project.peer_reviews_assignment_failed
  • project.peer_reviews_assigned
  • project.scoring_failed
  • project.scored
• Datamailer:
  • datamailer.outbox_enqueued
  • datamailer.outbox_acked
  • datamailer.outbox_dispatch_failed
  • datamailer.outbox_failed
  • datamailer.callback_received
  • datamailer.health_checked

Accounts and keys needed

Sentry:

• Create a Sentry project for this Django app.
• Set SENTRY_DSN.
• Optional: SENTRY_TRACES_SAMPLE_RATE, SENTRY_PROFILES_SAMPLE_RATE.
• Keep SENTRY_SEND_DEFAULT_PII=0 unless we explicitly review PII policy.

PostHog:

• Create a PostHog project.
• Set POSTHOG_API_KEY.
• Optional: POSTHOG_HOST if using EU/self-hosted PostHog instead of the default https://us.i.posthog.com.
• Use OBSERVABILITY_EVENT_BACKENDS=log,posthog or leave it unset after setting POSTHOG_API_KEY.

Healthchecks.io:

• Create one check for the Datamailer health command.
• Set HEALTHCHECKS_DATAMAILER_HEALTH_URL to that ping URL.
• Schedule uv run python manage.py monitoring_datamailer_health --json.

External uptime:

• No app key needed.
• Add /api/health/ and a couple of key public pages in UptimeRobot, StatusCake, or Sentry Uptime.

CloudWatch:

• No new app secret needed.
• Configure ECS log retention, alarms, and dashboard/metric filters in AWS.

Local testing

No external account is required locally.

Commands that passed:

uv run python manage.py test data.tests.test_observability
uv run ruff check course_management/observability accounts/signals.py accounts/views/impersonation.py data/tests/test_observability.py
uv run python manage.py check
uv run python manage.py test --timing --durations 30

Full test suite result before the test-log-noise cleanup:

• 701 tests passed.

The only warnings observed are existing django-allauth deprecation warnings.

Migration and cost-control notes

The event schema is provider-neutral and versioned with schema_version. Application code only calls record_event(...).

To switch later, for example from PostHog to Grafana/Loki/OpenTelemetry/S3:

1. Add a new EventBackend.
2. Configure dual-write, for example OBSERVABILITY_EVENT_BACKENDS=log,posthog,new_backend.
3. Compare event counts and payloads.
4. Remove posthog from the backend list when the replacement is verified.

This means increased traffic should not force a linear vendor cost increase. We can keep high-cardinality details in logs/event storage, keep CloudWatch metrics low-cardinality, sample if needed, and move product/audit events to a cheaper backend later without rewriting view/domain code.

[alexeygrigorev]

Implemented CloudWatch-only observability in two PRs:

• App PR: Keep observability on CloudWatch #215
• Infra PR: https://github.com/DataTalksClub/aws-infra/pull/14

What the app PR does:

• Keeps the app-owned observability abstraction: app code still calls record_event(...) / report_exception(...), not a vendor SDK.
• Removes old Sentry, PostHog, and Healthchecks provider code/settings/dependency.
• Adds a CloudWatch Embedded Metric Format backend that emits one low-cardinality metric: CourseManagement/App / AppEventCount, dimensions environment and event.
• Keeps rich fields like course/homework/project/release/distinct_id in CloudWatch Logs, not metric dimensions, to avoid high-cardinality custom metric cost.
• Adds middleware to report unhandled Django exceptions as an exception event.
• Updates Datamailer health to emit datamailer.health_checked and datamailer.health_warning when status is not OK.
• Documents local testing and rollout in docs/observability.md.

What the infra PR does:

• Adds SNS topic course-management-operator-alerts subscribed to Terraform variable alert_email.
• Adds CloudWatch alarms for dev and production app events: exception, datamailer.health_warning, Datamailer outbox failures, homework/project scoring failures, peer-review assignment failures, and API auth failure spikes.
• Adds ALB alarms for target 5xx, unhealthy targets, and p95 response time.
• Adds an EventBridge rule that runs production monitoring_datamailer_health --json every 15 minutes on ECS.

Accounts / keys needed:

• No Sentry, PostHog, Healthchecks.io, Grafana, or third-party observability account.
• Existing AWS account and GitHub Actions AWS credentials are enough for app deployment and Terraform.
• Confirm the SNS email subscription that AWS sends to alert_email after Terraform apply.

Required rollout step after merge/apply:

Use the existing ECS environment-variable update script to set these on live task definitions. Terraform includes these env vars, but container_definitions are ignored because CI/CD/env tooling owns task revisions.

Dev:

OBSERVABILITY_EVENT_BACKENDS=log,cloudwatch
OBSERVABILITY_ENVIRONMENT=dev
CLOUDWATCH_APP_METRIC_NAMESPACE=CourseManagement/App

Production:

OBSERVABILITY_EVENT_BACKENDS=log,cloudwatch
OBSERVABILITY_ENVIRONMENT=production
CLOUDWATCH_APP_METRIC_NAMESPACE=CourseManagement/App

Migration considerations:

• To migrate away from CloudWatch later, add another EventBackend and dual-write with OBSERVABILITY_EVENT_BACKENDS=log,cloudwatch,new_backend until counts match.
• App event producers do not need to change.
• Because metrics use only environment + event, cost should grow mainly with the number of distinct alarmed event types/projects, not with individual users, courses, homework IDs, or submissions.

[alexeygrigorev]

Review update after checking the change:

• Deployed ECS env should use OBSERVABILITY_EVENT_BACKENDS=cloudwatch, not log,cloudwatch. The CloudWatch backend already writes the structured event/EMF log record, so adding log would duplicate every event in CloudWatch Logs.
• App PR now sanitizes event property keys that collide with Python logging fields, for example name becomes app_name, so logging cannot drop events because of extra collisions.
• Infra PR no longer sends OK notification emails for alarm recovery; it only wires alarm notifications.

Rollout values remain:

Dev:

OBSERVABILITY_EVENT_BACKENDS=cloudwatch
OBSERVABILITY_ENVIRONMENT=dev
CLOUDWATCH_APP_METRIC_NAMESPACE=CourseManagement/App

Production:

OBSERVABILITY_EVENT_BACKENDS=cloudwatch
OBSERVABILITY_ENVIRONMENT=production
CLOUDWATCH_APP_METRIC_NAMESPACE=CourseManagement/App

[alexeygrigorev]

Closed: CloudWatch-only observability is implemented and merged.

Merged PRs:

• App: Keep observability on CloudWatch #215
• Infra: https://github.com/DataTalksClub/aws-infra/pull/14

Remaining operational rollout is tracked in the PR/issue comments: confirm the SNS subscription email and set the ECS env vars with the existing updater.

[alexeygrigorev]

Reopening for rollout verification.

Implementation is merged:

• App: Keep observability on CloudWatch #215
• Infra: https://github.com/DataTalksClub/aws-infra/pull/14

Still to verify operationally:

• Confirm the SNS subscription email for .

• Set live ECS task env vars with the existing updater:

  • /
• Deploy the app and confirm records appear in CloudWatch Logs.
• Confirm / metrics are created for at least one known event.
• Confirm one alarm path works, ideally with a controlled non-production event first.

After these are verified, this issue can be closed.

[alexeygrigorev]

Rollout verification status update.

Implementation is merged:

• App: Keep observability on CloudWatch #215
• Infra: https://github.com/DataTalksClub/aws-infra/pull/14

Still to verify operationally:

• Confirm the SNS subscription email for alert_email.
• Set live ECS task env vars with the existing updater:
  • OBSERVABILITY_EVENT_BACKENDS=cloudwatch
  • OBSERVABILITY_ENVIRONMENT=dev / production
  • CLOUDWATCH_APP_METRIC_NAMESPACE=CourseManagement/App
• Deploy the app and confirm cloudwatch_app_event records appear in CloudWatch Logs.
• Confirm CourseManagement/App / AppEventCount metrics are created for at least one known event.
• Confirm one alarm path works, ideally with a controlled non-production event first.

After these are verified, this issue can be closed.