upstate_phoenix

Copy Markdown View Source

Phoenix app health for Upstate: boot facts, pending migrations and Oban jobs. Counts only.

The package adds one Plug that answers GET /upstate/health. Upstate's host observer reads it from inside your server, over your private Docker network, every minute. Nothing else is installed, and nothing is sent anywhere by the package itself.

It is the Phoenix counterpart of the upstate-rails gem and answers the same schema 1 response, so Upstate needs nothing new on the server side.

What is sent

  • App version (APP_VERSION, else KAMAL_VERSION, else unknown), Phoenix, Elixir and OTP versions, the environment (prod), and when the VM booted.
  • The reporter name and version (upstate_phoenix, 0.1.0).
  • Whether every configured Ecto repo answers, and whether any of them has migrations that have not run.
  • Oban job numbers: queue depth (available, plus scheduled jobs whose time has come), age of the oldest of those, jobs completed and failed in the last 5 minutes, and when a job last ran.

What is never sent

No params, URLs or paths, SQL, IP addresses, user agents, headers, cookies, job arguments, job errors, queue names, worker names, migration names, exception classes or messages, logs, or traces. The Rails-only sections of the schema (rails, ruby, puma, active_record_pool, cache, actions) are left out.

Install

# mix.exs
def deps do
  [
    {:upstate_phoenix, "~> 0.1"}
  ]
end
# lib/my_app_web/endpoint.ex, above `plug Plug.Telemetry`
plug UpstatePhoenix.Plug, otp_app: :my_app

Placing it above Plug.Telemetry keeps the once-a-minute poll out of your request logs. Every other path passes straight through.

The endpoint is off until UPSTATE_HEALTH_TOKEN is set. Without it, /upstate/health returns 404. With a missing or wrong token it returns 401. Tokens are compared as SHA-256 digests in constant time.

force_ssl

The observer calls the container's private IP over plain HTTP, so force_ssl would answer with a redirect. Exclude the path:

# config/prod.exs (Plug 1.19+)
config :my_app, MyAppWeb.Endpoint,
  force_ssl: [
    rewrite_on: [:x_forwarded_proto],
    exclude: [hosts: ["localhost", "127.0.0.1"], paths: ["/up", "/upstate/health"]]
  ]

Phoenix compiles force_ssl into its endpoint. Set this in config/prod.exs and rebuild the release; changing only runtime.exs does not update the compiled SSL plug.

On Plug older than 1.19, whose :exclude takes only hosts, use exclude: {UpstatePhoenix, :private_host?, []} to skip the redirect for private-IP hosts.

Configuration

Everything has a default. Options can go on the plug call or in config, plug options first:

# config/config.exs
config :upstate_phoenix,
  repos: [MyApp.Repo],         # default: the otp_app's :ecto_repos
  oban: Oban,                  # the Oban instance name, or false for no job system
  environment: "prod"          # default: the Mix env the package was compiled in
  • repos entries can also be {MyApp.Repo, migrations_paths: ["priv/repo/migrations"]} when migrations live somewhere other than Ecto's default.
  • The token is read from UPSTATE_HEALTH_TOKEN at request time. You can also set config :upstate_phoenix, token: ... in config/runtime.exs.

Kamal secrets

  1. In Upstate, open the environment page and generate a health token. It is shown once.

  2. Add it to .kamal/secrets:

    UPSTATE_HEALTH_TOKEN=<the token>

  3. List it in config/deploy.yml:

    env:
  secret:
    - UPSTATE_HEALTH_TOKEN

  4. kamal deploy.

Verify

From the server (the observer uses the container's private IP and app port):

curl -i -H "Authorization: Bearer $UPSTATE_HEALTH_TOKEN" http://<container ip>:4000/upstate/health

You should get 200 and JSON with "schema": 1. A 503 means a database is not answering (the JSON says so). A section that fails reports null and the rest still answers.

Example response

{
  "schema": 1,
  "version": "4f2c1e7",
  "reporter": {"name": "upstate_phoenix", "version": "0.1.0"},
  "phoenix": {"version": "1.8.1", "elixir": "1.18.4", "otp": "27", "environment": "prod"},
  "booted_at": "2026-10-29T12:00:00Z",
  "database": {"ok": true, "pending_migrations": false},
  "jobs": {"adapter": "oban", "queue_depth": 0, "oldest_queued_age_seconds": null, "workers": null,
           "processed_last_5m": 42, "failed_last_5m": 0, "recent_processing_at": "2026-10-29T12:41:07Z"}
}

Good to know

  • Job numbers come from one aggregate query on oban_jobs through the running Oban instance's repo and prefix, so any web container reports the shared queue and job-only containers need no scraping.
  • failed_last_5m counts jobs discarded in the last 5 minutes plus retryable jobs whose failed attempt started in that window.
  • The counts only see rows still in oban_jobs. Oban.Plugins.Pruner defaults to keeping jobs for 60 seconds; keep them at least 5 minutes (max_age: 300 or more) or processed and failed counts will be low.
  • workers is always null: Oban has no per-node heartbeat table outside Oban Pro.
  • Oban on PostgreSQL is supported (Basic and Pro Smart engines). SQLite and MySQL engines report the adapter "unsupported" with no numbers. No Oban, or oban: false, reports "none".
  • The migration check reads schema_migrations without taking Ecto's migration lock and never creates the table, so it is safe while a migration runs.
  • Database calls time out after 1 to 1.5 seconds, inside the observer's 3-second budget.
  • Requires Elixir 1.15+ and Plug 1.14+. Ecto SQL and Oban are optional and used only when your app has them. JSON uses Elixir's built-in JSON (1.18+) or Jason.

Development

mix deps.get
PGUSER=postgres PGPASSWORD=postgres mix test

Tests create and migrate the upstate_phoenix_test database (override with UPSTATE_PHOENIX_TEST_DB, PGHOST).

License

MIT.

Health collection has a two-second total budget for database and Oban sections, collected concurrently. Migration reads have a separate 500ms budget, including connection checkout; missing evidence is reported as null. Cancelled jobs contribute their last attempt timestamp but not processed/failed counts.

An explicit migrations_paths: [] disables Ecto migration inspection and reports pending_migrations: null, retaining the database reachability check. Use this when another application (such as Rails) owns the schema.