Running Production Without an SRE Team: How AI Ops Replaced My On-Call

I run a production API with 434 endpoints, Mobile Money payments, real-time WebSocket chat, and thousands of users in Cameroon. The team is small. There's no dedicated SRE, no PagerDuty subscription, no Datadog bill. Instead, there's an AI operations platform that monitors, heals, and deploys autonomously via Slack.

This is how I replaced traditional on-call with 5 AI apps, 8 cron jobs, and AWS SSM as the control plane.

The Architecture

The system runs as a Docker container (Kiro Crew gateway) connected to Slack via Socket Mode:

Kiro Crew Container (port 5476)
|-- Gateway + Dashboard
|-- Slack bot (Socket Mode)
|-- Cron scheduler (8 jobs)
|-- 5 AI Apps:
    |-- ops-monitor     (infrastructure health + self-healing)
    |-- release-manager (deploy lifecycle via Slack)
    |-- pr-reviewer     (code standards enforcement)
    |-- db-manager      (migrations, seeds, backups)
    |-- user-support    (account lookups, payment checks)

The entire infrastructure it manages is a single EC2 instance (i-05d2778970da12fb1) in eu-west-1, accessed exclusively via AWS SSM (no SSH).

The 8 Cron Jobs

These are the heartbeat of the system. Each job fires on schedule, the AI evaluates the result, and either self-heals or alerts:

Job	Schedule	What it does
health-ping	Every 5 min	Pings API, auto-restarts if down
backup-check	Daily 10 AM	Alerts if backup older than 48h
disk-check	Every 6 hours	Auto-prunes Docker if disk > 75%
sentry-digest	Daily 8 AM	Summarises errors, auto-fixes obvious ones
weekly-report	Monday 9 AM	Full infra health report
dependency-scan	Wednesday 7 AM	Checks for vulnerable deps
flutter-analyze	Friday 7 AM	Mobile code quality trend
ssl-check	Monday 6 AM	Certificate expiry warning

Self-Healing Pattern: Detect, Fix, Verify, Report

The core pattern is consistent across all monitoring:

1. Detect: Cron fires, checks a condition (API responding? Disk under 75%? Backup fresh?)
2. Assess: Is this actionable? Has it been seen before?
3. Fix: Execute the least-disruptive fix first, escalate if needed
4. Verify: Confirm the fix worked
5. Report: Post to #alerts-sentry only if action was taken

Example: API Health Self-Healing

When the health-ping job detects the API is down:

Step 1: curl https://api.wefoundit.space/health (fails)
Step 2: Retry 3 times with 10s gaps (still failing)
Step 3: Restart api container via SSM:
        aws ssm send-command --instance-ids i-05d2778970da12fb1 \
          --document-name "AWS-RunShellScript" \
          --parameters 'commands=["cd /opt/wefoundit/wefoundit-api && docker compose restart api"]'
Step 4: Wait 30s, re-check health
Step 5: If still down, restart ALL containers
Step 6: Post result to #alerts-sentry with before/after status

Example: Disk Auto-Pruning

Step 1: Run "df -h /" via SSM send-command
Step 2: Parse percentage from output
Step 3: If > 75%:
        - docker builder prune -af
        - docker image prune -af
        - Report freed space
Step 4: If still > 85% after prune:
        - journalctl --vacuum-size=50M
        - Alert human with specific recommendations

AWS SSM as the Control Plane

Every infrastructure action goes through SSM send-command. The AI never SSH's into the box. This gives us:

• IAM authentication: Every command is tied to an AWS identity
• CloudTrail audit: Every command execution is logged
• No open ports: SSH port exists as fallback but is rarely used
• Timeout control: Commands have built-in timeout (30s for checks, 120s for deploys)

The command template used throughout:

aws ssm send-command \
  --instance-ids i-05d2778970da12fb1 \
  --region eu-west-1 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["export HOME=/root && <COMMAND>"]'

Then the system polls get-command-invocation for the result:

for attempt in range(10):
    result = ssm.get_command_invocation(
        CommandId=command_id,
        InstanceId=instance_id
    )
    if result["Status"] in ("Success", "Failed"):
        return result["StandardOutputContent"]
    time.sleep(3)

Sentry Auto-Fix: The Most Controversial Feature

The daily Sentry digest doesn't just report errors, it fixes obvious ones:

What it will auto-fix:

• Null/None checks (missing if x is not None guards)
• Missing imports
• UUID validation errors (wrong format passed to a function)
• Timezone-naive datetime comparisons
• Missing field validators in Pydantic schemas

What it will never auto-fix:

• Database migrations
• Payment or commission logic
• Authentication/security code
• Business rule changes
• Anything it's unsure about

The decision boundary is explicit in the agent's prompt: "If unsure, just alert #alerts-sentry and stop. Do not guess."

When it does fix something, the flow is:

1. Fetch stacktrace from Sentry API
2. Identify the error pattern (e.g., "AttributeError: 'NoneType' has no attribute 'id'")
3. Open the file, apply the fix (add null check)
4. Validate: python -m compileall <file>
5. Commit + push to main
6. Deploy via SSM (docker compose rebuild)
7. Mark issue as resolved in Sentry
8. Post to #alerts-sentry: "Fixed [issue], deployed, resolved"

Is this scary? A little. But the guardrails make it safe:

• Only fixes patterns it's been explicitly trained on
• Always validates before committing (compile check)
• The "never auto-fix" list is a hard boundary
• Every action is posted to Slack where a human can immediately revert

Release Gate: AI-Powered Deploy Safety

Before any production deploy, the release-manager runs a 4-stage gate:

Stage 1: python -m compileall (syntax check all files)
Stage 2: pre_release_check (33 assertions about code quality)
Stage 3: test_user_endpoints (42 API endpoint smoke tests)
Stage 4: test_all_user_endpoints (exhaustive endpoint sweep)

Deploy is triggered via Slack: deploy api. The AI runs the gate, and only if all stages pass does it execute:

# Via SSM send-command
cd /opt/wefoundit && sudo bash wefoundit-api/deploy/scripts/update.sh

The update script: pulls latest code, rebuilds containers, runs migrations, verifies health.

What This Costs

Component	Monthly Cost
Kiro Crew (local Docker)	$0
Slack (free tier)	$0
Sentry (free tier, 5k events)	$0
AWS SSM send-command	$0
S3 backup storage	< $0.10
Total monitoring cost	~$0.10/month

Compare that to PagerDuty ($21/user/month), Datadog ($15/host/month), or even a basic uptime monitor ($10/month). The AI ops approach costs effectively nothing because it runs on existing free-tier services.

What's Missing (Honest Gaps)

1. No rollback automation: If a deploy breaks, the system detects it (health-ping) and restarts containers, but doesn't roll back to the previous code version.
2. Single-instance SPOF: The Crew container itself isn't monitored by anything external. If it dies, all monitoring stops.
3. No memory/CPU monitoring: Disk is checked, but RAM and CPU are not. OOM kills would only be caught after the fact via health-ping.
4. No rate limiting on fixes: The Sentry auto-fixer could theoretically deploy multiple times in rapid succession if many errors arrive simultaneously.

These are known tradeoffs for a small team. Each would be the next thing to solve as the platform scales.

When This Approach Works

• Small teams (1-5 engineers) running production services
• Services deployed on EC2/VMs where you control the host
• Environments where AWS SSM is already available (IAM roles attached)
• Teams that use Slack as their primary communication tool
• Applications with well-defined health checks

When It Doesn't

• Large teams that need proper incident management workflows
• Multi-region deployments requiring coordinated failover
• Compliance environments that require human approval for all changes
• Systems where auto-fixing code is unacceptable (regulated industries)

Key Takeaways

1. AI ops isn't about replacing humans, it's about handling the obvious stuff. The 3 AM restart, the disk cleanup, the null-check fix. Humans handle architecture decisions and ambiguous problems.
2. SSM send-command is the underrated backbone. It gives you remote execution with IAM auth, CloudTrail logging, and no open ports. Perfect for AI-driven automation.
3. The "never auto-fix" list is more important than the "auto-fix" list. Define your boundaries explicitly. The AI should know what it cannot touch.
4. Slack as the single pane of glass works for small teams. Every alert, every deploy, every fix proposal lands in one place where the team can see it.
5. This costs almost nothing. The combination of free-tier services (SSM, S3, Slack, Sentry) means AI ops is accessible to bootstrapped startups, not just well-funded platforms.

---

The WeFoundIt Crew platform runs on Kiro Crew with custom apps for each operational domain. The full architecture is documented in the wefoundit-crew repository.