Where a request waits on its way to a worker

When request workers are on (workers.requestCount above 0), the process that owns the listeners, the front process, does not answer most HTTP requests itself. It hands each one to a worker thread (Architecture describes the pools). This page follows one request from the socket to the worker and back, names every place it can wait, and explains the log line that measures each wait.

A request passes the main port's listen backlog; in the front process, the ticket wait, the barrier and the per-worker connection pool; in a request worker thread, the socket backlog, the event loop, the handler and the commit to PostgreSQL; then the answer returns to the client. Brackets above the stages name the phase of the request phases log line that measures each.

The path, in order

# Where it waits What it is waiting for What bounds it
1 The main port’s listen backlog The front process’s event loop to accept the connection The operating system’s backlog
2 The front process: ticket Writes made by earlier requests through other workers to be announced as committed 2 seconds, then the request is served without them
3 The front process: barrier The chosen worker to catch up with those writes 5 seconds
4 The front process: the worker’s connection pool A free connection to that worker workers.maxSockets connections per worker (default 64); requests past it wait their turn in the front process’s memory
5 The worker’s socket backlog The worker to accept the connection A backlog of 1,024 on the worker’s unix socket
6 The worker’s event loop The worker’s one thread to be free of the requests before it Nothing; this is the real queue
7 The worker: commit The request’s own writes to commit to the database before the answer is sent The database

Stages 2 and 3 are the read-your-write wait; Read-your-write below says when a request has to do them. Stage 4 is connect. Stages 5 and 6 are loop: the front times from the socket it was given until the worker says its middleware started. Stage 7 happens after the handler has run; a request that wrote nothing skips it.

Some paths never go to a worker. GET /healthcheck is answered by the front process at once, so a liveness probe waits for none of the above. A few console pages that read or change what only the front process holds are always answered there too: listeners, the TLS trust store and connection, the debugger, the SPIFFE brokers, the scheduler, the worker pools and node health.

Which worker

A request carrying a session cookie, the pool’s own pin cookie or a sign-in flow in its address (authn, consent, device_code, SAMLart) goes to the worker that served that key before, so one browser’s sign-in stays on one worker. A request carrying none of them, such as a token request from a client’s server, goes to the worker with the fewest requests in flight.

Bulk traffic has its own lane. Paths listed in workers.batch (by default /scim and the two Shared Signals receivers) are sent only to a share of the workers (workers.batchWorkerShare, default 50%), and only workers.batchConcurrency of them (default 8) are in flight at once. The rest wait in a list of at most workers.batchQueueLimit (default 5,000) for at most workers.batchQueueTimeoutS (default 60 seconds). That way a bulk load cannot fill every worker’s event loop, and sign-ins go on being answered beside it.

Read-your-write

Workers share nothing but the database, and a worker learns of another worker’s writes within about half a second to a second. Without a wait, a browser could sign in through one worker and be sent to another that has not yet heard of the session. workers.readYourWrite decides what a request waits for:

Value A request waits for
session (the default) The writes of earlier requests that carried the same session cookie, pin cookie or flow, every write of a request that carried none of them (a relying party’s server, an API caller, a dispatched LDAP operation), and the front process’s own writes. It does not wait for another browser’s writes. A code exchange at the token endpoint waits for its code’s writes, and a refresh grant that names one refresh token waits for that token’s writes, as below. Any other request carrying none of them waits as in global.
global Every write any request made before it, through any worker.
off Nothing. Use it only where every client is pinned to one worker.

true means session and false means off.

This setting is about the workers inside one node. Between the nodes of a cluster, a request catches up with other nodes’ writes in its own way: How the cluster works.

In session mode the token endpoint’s code exchange has no session cookie, because it comes from the client’s server rather than the browser. It waits for its CODE instead. When the authorization endpoint redirects the browser with code= in the query, the pool files that answer under the code, together with whatever the browser’s session and sign-in had still outstanding. A POST to the token endpoint with a form body of at most 16 KiB and a declared length is read before it waits, and passed to the worker unchanged. If it is an authorization_code grant for a code the pool learnt that way, it waits only for those writes.

A refresh grant in that same small form, naming one refresh token, waits for that token instead of for every write (#879). The token response is read when it is a 200 JSON body of a declared length of at most 64 KiB, and the new refresh token is filed before those bytes are sent, so the refresh that follows waits for the write that minted it. A token this process has not seen waits only for that token’s key — empty when no rotation of it is unfinished here — and for keyless writes. A token response this process could not read is one of those keyless writes. The refresh’s own write is not a keyless one, so another refresh and a browser do not wait for it. 8081 was not re-measured.

A refresh grant, and any other token request keyed by its code or by a browser’s cookie, also waits for every write that ended what a refresh token rests on: a sign-out, a global logout, a token or grant revoked, a refresh family revoked, a disabled account. Such a write is recognised by its path (a sign-out door, a write to /oauth2/revoke, /admin-api, /account-api, /scim or /ssf, any DELETE) or because the worker that answered it says so on its answer. So a refresh sent just after a sign-out is never answered by a worker that has not applied it. A browser’s ordinary request does not wait for these writes. stats().sessionBarrier.revokingDispatched, revokingAnswered and revokeWaits count them. The pool keys a refresh token by its SHA-256, never by the token itself.

Everything else at the token endpoint waits as in global: a code the pool did not learn (one delivered by form_post or in a fragment, or minted on another node), a refresh grant this process cannot read (more than one token, a token too long to be a key, or a body that is larger, chunked or sent over HTTP/2 without a length). stats().sessionBarrier.codeKeyed counts the exchanges keyed by their code, and refreshKeyed the refresh grants keyed by their token.

Reading the request phases log line

Once a minute each process logs, at info, its twelve busiest routes of the last minute with the 50th and 95th percentile of each phase in milliseconds:

request phases, last 60 s: POST /oauth2/token 32.8/s — p50/p95 ms:
  ticket 0/1, barrier 70/308, connect 1/8, loop 9/64, handler 93/256,
  commit 59/195, bridge 6/35, total 267/653

The figures above show the shape of the line. They are not a measurement of 8081, and 8081 was not re-measured when queue was split.

Phase Measured by What it is
ticket The front process Stage 2
barrier The front process Stage 3
connect The front process Stage 4: from asking the per-worker agent for a socket until it gives one (workers.maxSockets)
loop The front process, from an instant the worker reports Stages 5 and 6: from that socket until the worker’s middleware started
handler The worker The request’s own work, from reaching the worker’s middleware to its answer being ready
commit The worker Stage 7
bridge The worker Time inside handler that the worker’s thread was blocked in synchronous database reads. It is part of handler, not added to it.
total The front process From the request reaching the front process to the answer’s headers coming back from the worker

The phases do not add up to total. bridge is inside handler, and the short trip that carries the answer’s headers back to the front is in total only.

A route is the method and the first two path segments, without a realm prefix or a query. The route is not a request count: GET /oauth2/authorize appears twice in one OpenID Connect sign-in.

Three more kinds of line describe writing to the database rather than a request, and appear in the process that does the writing:

Line What it measures
FLUSH minted One write of minted state (sessions, codes, tokens): prepare (sealing the rows), db (the database’s time), rows, total
FLUSH directory One write of directory changes: rows, total
COMMIT through One response’s commit wait (stage 7), split into the directory, minted and keys writes it waited on. The parts run at the same time, so each is the time from the start of the wait until that part was done. The largest is the one that held the response, and they do not add up to total. A commit that failed is not counted.

The refresh ceiling

A successful rotating refresh waits for one synchronous commit: the minted flush of the new refresh token. The spend of the presented token commits without waiting for the log flush, and that later commit flushes the spend before the response is sent. The refresh token’s own write stays synchronous. A crash before the flush returns loses a token that was not issued.

Postgres flushes the log one fsync at a time, so that one flush is the ceiling. On the soak machine pg_test_fsync measured fdatasync at about 453 a second (2.2 ms). Refreshes topped out at about 155 to 180 a second while the spend and the token were two synchronous commits. commit_delay at 500 µs was slower there, and the service does not set it. 8081 was not re-measured after the spend stopped waiting.

What to look at first

  • loop high at p50. The workers’ threads are busy, or their listen backlog is. Look at handler and bridge for the routes that dominate, and at whether there are cores to spare for another worker (Monitoring → Worker Pools).
  • loop low at p50 and high at p95. Bursts are stacking behind slow requests on one worker. Password hashing and bulk imports are the usual causes; workers.batch keeps a bulk load in its own lane.
  • connect high. Requests are waiting for a free connection (workers.maxSockets) before a worker sees them.
  • barrier or ticket high. Requests are waiting for other workers’ writes. Check that workers.readYourWrite is session. At the token endpoint, compare stats().sessionBarrier.codeKeyed and refreshKeyed with global: a code exchange keyed by its code waits only for its code, a refresh grant keyed by its token waits for that token, for keyless writes and for writes that ended a session, token, family or account, and an unknown code or a large body still waits as in global, as described above.
  • commit high. The database is slow to commit. Compare COMMIT through’s parts with the FLUSH lines: a part much longer than its flush is waiting for an earlier write to finish first. A rotating refresh’s remaining wait is that one flush (The refresh ceiling).
  • Raising workers.maxSockets is rarely the answer. It moves the wait from connect (stage 4) to loop (stage 6) without making the worker faster. Do it when connect is the high phase and loop is not. Do not make it small either: the service makes requests to itself, and a worker whose connections are all held by requests waiting on such a call needs one more to make progress.
  • Architecture: the pools, and what runs where.
  • Configuration: every workers.* setting with its environment variable and default.
  • Persistence: what is written to the database, and when.
  • How the cluster works: the same questions between nodes.
  • Monitoring → Worker Pools (/admin/worker-pools): each worker’s requests in flight and served, live.