TL;DR

• On 64-day staging certificates, Let's Encrypt's ARI endpoint suggested renewing between day 42.04 and 43.32, 65.7 to 67.7 percent of the lifetime. We measured the same share at 45 and 90 days. That is about 21 days before expiry.
• certbot 5.8 renewed inside that window (it picks a random time in it on every run), acme.sh 3.1.6 scheduled day 42.16, and lego 5.5.2 allowed renewal from day 42.67. These configurations are ready for shorter lifetimes.
• certbot 2.9, the version apt installed on our Ubuntu 24.04 machine, does not use ARI and became due 30 days before expiry: day 34 of 64. Early, but safe.
• acme.sh 3.0.7 schedules the next renewal at issue + --days - 1 day, with no check against the expiry date. Its default of 60 gave day 59.04 of 64, and --days 80 gave day 79.04. Every acme.sh tag we scanned from 3.0.7 to 3.1.2 has the same default of 60.
• Search your cron jobs, wrapper scripts and monitoring for hardcoded 60, 80 and 83, as Let's Encrypt itself suggests, and still check your renewal timer, challenge path and deploy hook.

Prerequisites

• A server that renews certificates with certbot, acme.sh or lego
• Shell access to check the client version and its stored renewal settings
• To reproduce the test: a throwaway VM with port 80 reachable, and faketime

Everything we ran, and every output file, is in the repository:

The-DevOps-Daily/le-64-day-renewal-check on GitHub

What changes, and when

From Let's Encrypt's announcement of 7 October 2026:

• 14 October 2026: staging switches to 64-day certificates. Our watcher's request at 21:25 UTC still returned a 90-day certificate, and the one at 21:35 UTC returned a 64-day certificate.
• 10 February 2027: all subscribers move to 64-day certificates by default. The last 90-day certificates expire around 11 May 2027.
• The authorization reuse period drops from 30 days to 10 days, and to seven hours in 2028.
• 2028: 45-day certificates by default.

The advice in the announcement is short: if your renewals are automated and your client supports ARI, you should be all set; if not, renew at about two thirds of the lifetime, and look for hardcoded values such as 83, 80 or 60 in cron jobs, wrapper scripts and runbooks. We wanted to see which clients actually follow that, and what happens to the ones that do not.

How we measured it

On a throwaway Ubuntu 24.04 droplet we installed five client versions side by side: certbot 2.9.0 from apt, certbot 5.8.0 from snap, acme.sh 3.0.7 and 3.1.6 from the upstream repository, and lego 5.5.2. They requested real certificates from the Let's Encrypt staging environment, using hostnames under sslip.io that resolved to the droplet, at three lifetimes. Where a client could not request a lifetime, we used copied state or its source:

• 64-day certificates from the default classic profile, after staging switched. A watcher asked for a classic certificate every 10 minutes from 12:36 UTC and started the suite as soon as the lifetime dropped.
• 90-day certificates from the same classic profile, earlier that evening.
• 45-day certificates from the tlsserver profile, the lifetime planned for 2028. acme.sh 3.0.7 and certbot 2.9 cannot request a profile, so certbot 2.9 was tested on a copy of certbot 5.8's 45-day certificate and acme.sh 3.0.7 was read from its source.

Then, for each certificate:

• acme.sh stores its planned renewal time in its config, so we read it.
• lego prints, at issuance, when a renewal will be allowed; we ran it with ARI on and with --ari-disable.
• certbot only decides when certbot renew runs, so we ran it under faketime at chosen hours after issuance, each time on a fresh copy of the state, and recorded whether it renewed.
• A small script asked the ARI endpoint (RFC 9773) for the suggested window of every certificate.

Days are counted from each certificate's notBefore time, which Let's Encrypt sets about an hour before issuance. This is the certbot 2.9 probe on the 64-day certificate, as recorded:

certbot 2.9.0 under faketime, 64-day staging certificate

$ probe-certbot.sh certbot $R/cb29 c29n.46-101-102-127.sslip.io 800 812 815 818 830
# certbot 2.9.0 c29n.46-101-102-127.sslip.io notBefore=Oct 14 20:37:00 2026 GMT notAfter=Dec 17 20:36:59 2026 GMT
hour 800 (day 33.33, 2026-11-17 04:37:00 UTC): not due
hour 812 (day 33.83, 2026-11-17 16:37:00 UTC): not due
hour 815 (day 33.96, 2026-11-17 19:37:00 UTC): not due
hour 818 (day 34.08, 2026-11-17 22:37:00 UTC): RENEWED
hour 830 (day 34.58, 2026-11-18 10:37:00 UTC): RENEWED

Not due at day 33.96, renewed at day 34.08: the threshold is between the two, consistent with 30 days before the 64-day expiry.

Results

Day each client renews, or plans to renew, a 64-day certificate

The certificate expires on day 64. Measured on Let's Encrypt staging on 2026-10-14, one certificate per row. certbot 2.9 became due between day 33.96 and 34.08. certbot 5.8 is its first renewal in our probes; it picks a random time inside the ARI window (day 42.04 to 43.32) on every run. acme.sh shows its stored schedule and lego the day it reported at issuance. The last bar is the --days 80 certificate after upgrading acme.sh to 3.1.6. Source: results/measured-64.txt in the repo.

ARI clients: the window does the work

The ARI window opened at 65.7 percent of the lifetime and closed at 67.7 percent at all three lengths: day 42.04 to 43.32 of 64, day 29.56 to 30.46 of 45, and day 59.11 to 60.91 of 90. On a 64-day certificate, that is 21 to 22 days before expiry. A client that follows the window needs no change for 45 days in 2028 either.

While we waited, we projected the 64-day column from the 45-day and 90-day runs and committed the projection at 21:32 UTC (commit f909cfc in the repository). Our next request, at 21:35 UTC, returned the first 64-day certificate. Every 64-day result in the table matches that projection (results/projection-64.txt and results/measured-64.txt).

certbot 5.8 does not renew at one fixed moment. Each time it runs, it asks for the window, picks a random time inside it, and renews if that time has passed. From its source (acme/client.py):

delta_seconds = (end - start).total_seconds()
random_seconds = random.uniform(0, delta_seconds)
random_time = start + datetime.timedelta(seconds=random_seconds)

So a run a third of the way into the window renews about a third of the time. These are our 64-day probes, each from a fresh copy of the same state:

hour 1010 (day 42.08, 2026-11-25 22:37:12 UTC): not due
hour 1014 (day 42.25, 2026-11-26 02:37:12 UTC): not due
hour 1020 (day 42.50, 2026-11-26 08:37:12 UTC): RENEWED
hour 1030 (day 42.92, 2026-11-26 18:37:12 UTC): not due

The run at day 42.50 had about a 36 percent chance and renewed. The run at day 42.92 had about 68 percent and did not. This spreads renewals out, as RFC 9773 intends, so a fleet does not renew all at once. Once the window has closed, every run renews: on the 45-day and 90-day certificates, every probe after the window renewed. If the ARI request fails, certbot 5.8 catches the error and gets no ARI time; it then renews at two thirds of the lifetime (day 42.67 of 64), or by renew_before_expiry if that is set in the renewal config. We read this in its source (results/certbot-5.8-ari-source.txt) but did not test a failed ARI request.

acme.sh 3.1.6 scheduled its next renewal for day 42.16, with its default and with --days 80. It also stayed inside the window with --days 60 at 45 days and with --days 80 at 90 days. lego 5.5.2, checked at issuance, gave the same answer with ARI on and off: renewal allowed from day 42.67, exactly two thirds of the lifetime.

These are the clients' decisions, not your whole setup. A renewal still needs a timer or cron job that runs often enough, a working challenge path and a deploy hook that reloads the server.

certbot 2.9: early, and fine

certbot added ARI support in version 4.1.0 (June 2025). The certbot that apt installed on our Ubuntu 24.04 machine was 2.9.0, and its probes became due 30 days before expiry at every lifetime: between day 33.96 and 34.08 of 64, between day 14.96 and 15.08 of 45, and between day 59.79 and 60.21 of 90. It renews more often than it needs to, and it leaves about 30 days to notice and fix a failed renewal, if its timer runs normally.

acme.sh 3.0.7: the one that breaks

acme.sh 3.0.7 computes the next renewal when it issues a certificate, and never compares it with the expiry date. From its source:

Le_NextRenewTime=$(_math "$Le_CertCreateTime" + "$Le_RenewalDays" \* 24 \* 60 \* 60)
Le_NextRenewTime=$(_math "$Le_NextRenewTime" - 86400)

Le_RenewalDays is the --days value, 60 by default. The 64-day staging run shows the result:

acme307 a307n notBefore=Oct 14 20:37:26 2026 GMT notAfter=Dec 17 20:37:25 2026 GMT next='2026-12-12T21:36:01Z' days=
acme307 a307n80 notBefore=Oct 14 20:37:43 2026 GMT notAfter=Dec 17 20:37:42 2026 GMT next='2027-01-01T21:36:18Z' days='80'

Both certificates expire on 17 December. With the default, the next renewal is 12 December, day 59.04, so one failed renewal (a DNS hiccup, a firewall change, a rate limit) leaves very little time before expiry. With --days 80, the next renewal is 1 January 2027, day 79.04, fifteen days after the certificate expired, and acme.sh 3.0.7 does not try before then. The 90-day run gave the same days, 59 and 79, which were safe at that lifetime. At 45 days in 2028, the default fails too.

We scanned each acme.sh tag from 3.0.7 to 3.1.6 for the scheduling code (the scan is in the repo):

We only ran 3.0.7 and 3.1.6. The rows for the versions in between come from reading the source, so check your own version's behaviour if it is one of those.

What upgrading does to an existing certificate

acme.sh stores Le_RenewalDays and the next renewal time in each certificate's config, so an upgrade does not by itself change a schedule written by an old version. We tested what happens next: we copied the state of the 64-day certificate that 3.0.7 had scheduled for day 79 and ran acme.sh 3.1.6 --renew on it, without --force:

== before: acme.sh 3.0.7 state
Le_RenewalDays='80'
Le_NextRenewTimeStr='2027-01-01T21:36:18Z'
notAfter=Dec 17 20:37:42 2026 GMT
== acme.sh 3.1.6 --renew on the same state (no --force), 2026-10-14T23:22:34Z
[Wed Oct 14 23:22:34 UTC 2026] Renew: a307n80.46-101-102-127.sslip.io
[Wed Oct 14 23:22:34 UTC 2026] Renewing: 'a307n80.46-101-102-127.sslip.io'
[Wed Oct 14 23:22:34 UTC 2026] Renewing using Le_API=https://acme-staging-v02.api.letsencrypt.org/directory
[Wed Oct 14 23:22:35 UTC 2026] Current renewal time: 2027-01-01T21:36:18Z
[Wed Oct 14 23:22:35 UTC 2026] ARI suggestedWindow: 2026-11-25T21:36:11Z to 2026-11-27T04:18:13Z
[Wed Oct 14 23:22:35 UTC 2026] Updating renewal time picked from ARI window: 2026-11-26T02:10:18Z
[Wed Oct 14 23:22:35 UTC 2026] Skipping. Next renewal time is: 2026-11-26T02:10:18Z
== after
Le_RenewalDays='80'
Le_NextRenewTimeStr='2026-11-26T02:10:18Z'

The stored --days 80 stays, but 3.1.6 asked the ARI endpoint on its first check and moved the renewal from 1 January, after expiry, to 26 November, day 42.23. The same test on the 90-day certificate gave the same result. Its source says it does this on every renewal check unless NO_ARI=1 is set. On a CA without ARI, or with NO_ARI=1, the stored schedule is what counts.

What to check on your servers

Find out which client and version renews each certificate, and what it has stored:

# certbot: version, and any renewal setting that overrides the default
certbot --version
grep -H renew_before_expiry /etc/letsencrypt/renewal/*.conf

# acme.sh: version, the stored --days, and the stored next renewal time
~/.acme.sh/acme.sh --version
grep -H -E '^Le_RenewalDays=|^Le_NextRenewTimeStr=' ~/.acme.sh/*/*.conf

# lego: version (ARI is on unless you pass --ari-disable)
lego --version

Then:

1. Upgrade acme.sh to 3.1.6 or later wherever it runs (acme.sh --upgrade), and run acme.sh --cron once to see the new schedule. On anything that has to stay on an older version, compare each stored Le_NextRenewTimeStr with the certificate's expiry, and lower --days so the next renewal comes well before it.
2. Look for 60, 80 and 83 in cron jobs, systemd timers, wrapper scripts and runbooks, as Let's Encrypt suggests. A script that renews "when the certificate is 60 days old" leaves four days of margin at 64 days and none at 45.
3. Check your expiry alerts. The ARI clients chose renewal times with about 21 days left on our 64-day certificates, so an alert at 30 days left fires on every certificate while renewal is working fine, and people learn to ignore it. Alert on failed renewals, and on expiry with a threshold below the normal renewal point, for example 14 days left.
4. Check what else issues certificates. The shorter authorization reuse period (10 days instead of 30) does not change a renewal every 42 days, since both periods have passed by then. It matters when you issue again 10 to 30 days after the last validation, for example a manual reissue or a renewal forced early: that order now needs a fresh challenge.
5. Rehearse on staging before February. Staging's classic profile already issues 64-day certificates, and its tlsserver profile issues 45-day ones. Issue a staging certificate with separate state from production and run your normal renewal and deploy path against it.

What this test does not show

• One certificate per configuration. We issued one 64-day certificate for each client setup, on one evening. certbot 5.8 picks a random time inside the window, so our probes show that it renews inside the window, not how its renewals spread across a fleet.
• ARI windows can move. Let's Encrypt can change a certificate's window, for example to ask for early renewal before a mass revocation. Clients that check ARI on each run, as certbot 5.8 and acme.sh 3.1.6 do, pick up the change; we did not test that. We measured the normal window. The acme.sh 3.0.7 and certbot 2.9 rows would not change, because neither looks at ARI.
• acme.sh 3.0.7 at 45 days comes from its source. It cannot request the 45-day profile, so we could not issue one with it. The 64-day and 90-day runs confirm the formula.
• lego was checked at issuance only. We recorded when it said a renewal would be allowed, not a renewal at that time.
• Five client installs. We did not test Caddy, Traefik, cert-manager, win-acme or hosting panels that embed their own ACME code. Check those against the same two questions: does it use ARI, and if not, how does it decide?
• Staging, not production. Production should behave the same way from February, but we only issued staging certificates.

The short version: in our tests, the clients with ARI enabled renewed on Let's Encrypt's window, and that window was the same share of the lifetime at 45, 64 and 90 days. If it does not, find out what number it renews on, because that number was chosen for 90 days.

So on this test the cheap model was good enough. That is not the whole answer. DeepSeek V4 Flash took 4.4 seconds at the median and once took a full minute. Two other cheap models each let through alerts the policy says must page. And 200 clean results still leave room for a real error rate of 1.5 percent. The most useful thing we learned came from a bug in our own test data: most of the models found it, and our second reviewer did not.

TL;DR

• With a clear written policy, DeepSeek V4 Flash, Qwen3.8-Max and router:general made the policy's call on all 600 alert runs. gpt-oss-120b got 599 (one empty reply). Qwen3.8-Max costs about ten times as much per alert as DeepSeek V4 Flash, and it answered about three times faster.
• router:general sent 571 of 600 calls to Qwen3.8-Max, so it cost almost what Qwen3.8-Max costs. router:software-engineering sent most calls to DeepSeek V4 Flash and got 4 wrong, all 4 from a model it reported as deepseek-4-flash.
• Llama 4 Maverick failed to page on 2 different page-level alerts (3 times in all) and gpt-oss-20b on 3 (once each). Ministral 3 14B missed 4.7 page-level alerts per run out of 50, and 17 of its 600 replies contained two different decisions.
• DeepSeek V4 Flash spends most of its output on reasoning: p95 latency of 10.7 seconds, 33 calls over 10 seconds and one at 60.6 seconds.
• At 10,000 alerts a month, the token cost is about $1.57 for the cheap model and $16.40 for the expensive one. Choose on missed pages and latency, not on price.

Prerequisites

• A DigitalOcean account with a Serverless Inference model access key
• Node.js 20 or later (the scripts have no dependencies)
• About $3 of inference if you rerun all eight configurations

The code, the alerts, every recorded decision and reply, and the scoring are in the repository:

The-DevOps-Daily/alert-triage-bench on GitHub

The task: one policy, three answers

The simplest way to route alerts is the severity label. Whoever wrote the alert rule decided that KubePodCrashLooping is a warning and EndpointDown is critical, and the route pages on critical. That works until the label and the situation disagree: a critical alert from a sandbox, a warning on a checkout outage, a probe that still fires for a service you turned off last month.

The question here is narrower than "can a model run on-call". Give a model the alert and a written policy, and ask it to make the one call a human makes first. The policy is short on purpose, the kind a team could actually keep up to date. In summary:

Page    production, and customer impact now or within 1 hour:
        a tier 1 service down, >2% errors for 5+ min or p99 >2 s for 10+ min,
        a tier 1 SLO burning at 14x or faster,
        a database, cache or queue full within 1 hour,
        a public cert expiring in under 24 h, the primary DB unreachable,
        confirmed unauthorized access
Ticket  a real problem with no customer impact within the hour:
        tier 2 problems, tier 1 problems below the page thresholds,
        capacity gone in more than 1 hour but less than 7 days,
        certs expiring in 1 to 14 days,
        one pod or node failing while the service is fine,
        failed backups and jobs, staging problems that block a release
Ignore  non-production (unless it blocks a release),
        alerts that resolved within 5 minutes with no customer impact,
        heartbeats, deploy and autoscaling notices,
        maintenance windows and decommissioned or test targets
        (these two win even over a page)
        The severity label is a hint only.

The full text, including the list of tier 1 services, is in POLICY.md. Every model got the whole policy as its system prompt and one alert as the user message, about 720 input tokens per call, and was told to reply with one JSON object, {"decision": ..., "reason": ...}, and nothing else.

A typical hard case looks like this. The service is tier 1 and the restart count is alarming, but one pod out of eight is not an outage:

status: firing
labels: alertname=KubePodCrashLooping env=production service=search severity=warning team=discovery
summary: Pod search-ad1 is crash looping
description: 1 of 8 search pods has restarted 21 times in the last hour (OOMKilled). The other 7 replicas are healthy and the service error ratio is 0.16%.
startsAt: 2026-10-15T03:41:00Z
maintenance window: none

The answer is ticket. So is a staging outage that blocks a release, even though staging is otherwise ignored. A severity=critical alert showing 42 percent errors on search in staging, with no release waiting, is ignore.

How the 200 alerts were built and checked

The alerts are generated, not collected. Real alert payloads would not come with a known correct answer, and hand-labelling 200 of them would test our labelling as much as the models. Instead, a seeded script uses 31 scenario templates. Each template has the decision the policy gives it, and draws its numbers (service, error rate, minutes firing, hours to expiry) from ranges that keep that decision. The text the models see is rendered from the same fields. Writing templates by hand can put mistakes in the text or the label, and one of ours did, as you will see.

alert-triage-bench

$ node scripts/01-generate-alerts.mjs
alerts: 200
  page   50 (hard 14)
  ticket 84 (hard 40)
  ignore 66 (hard 54)
  severity label points the wrong way: 98
$ node scripts/03b-baseline-severity.mjs
severity-label baseline: 102 of 200 correct

Two numbers in that output need explaining. "Hard" marks alerts where a shortcut gives the wrong answer: the severity label points the wrong way, or the alert name sounds worse than the policy says it is. And 98 of the 200 severity labels are wrong on purpose. That makes the no-model baseline deliberately weak: it scores 51 percent and lets 14 of the 50 page-level alerts through. It measures whether a model reads the alert instead of the label. It says nothing about how a model compares with a well-written rules engine, which we did not test.

To check that the text carries what the labels assume, we gave one AI second reader, Codex, only POLICY.md and the 200 rendered alerts, never the labels, and asked it to label them. It agreed on all 200. Hold on to that number, because it turned out to be wrong in an interesting way.

Results

Eight configurations, three runs each, 4,800 calls. The six direct models are open-weight models. Our account's key returned 403 this model is not available for your subscription tier for the GPT-5 and Claude models we tried, so they are not in this test (the check is saved in the repo). The two routers are DigitalOcean's Inference Router, in public preview, which picks a model per request and reports which one it used.

Costs are estimates: recorded token usage times DigitalOcean's list prices on 2026-10-14. One model the router reported, deepseek-4-flash, is not on the price page under that name, and we priced it as DeepSeek V4 Flash.

Wrong answers out of 600 (200 alerts, 3 runs)

Lower is better. An empty reply counts as wrong. Ministral 3 14B got 68 of 600 wrong and is left out so the rest stay readable. Source: results/summary.json in the repo.

Estimated cost per 1,000 alerts

Recorded token usage times DigitalOcean list prices checked 2026-10-14. A routed call is billed at the price of the model that served it; the router itself is free during the public preview.

Three configurations scored 600 of 600, four more got between one and five wrong, and one is clearly not good enough. Ministral 3 14B is the fastest and among the cheapest, but it got 42 of the 200 alerts wrong in at least one run, missed 4.7 page-level alerts per run and gave the same answer in all three runs for only 80 percent of the alerts. It also ignored the format: 593 of its 600 replies had text around the JSON, and 17 contained two different decisions, typically an answer followed by a "correction".

That raises a scoring question, so here is the rule: the scorer takes the first decision in the reply, because an automated router would act on the first answer it can parse, and an empty reply counts as wrong. The repo records the first 400 characters of each reply from this run. The runner now keeps the full text for future runs.

The misses from the near-perfect models are worth reading, because they are the errors that hurt. Llama 4 Maverick called a Postgres database that would run out of connections in about 34 minutes a ticket, twice. Once it answered ticket for a public certificate expiring in 16 hours, then wrote "Wait, the correct response should be based on the given policy" and started a second answer, which we score as the ticket it gave first. gpt-oss-20b let three different Redis memory alerts through, all in its second run. Each of those replies would have suppressed a page that the policy requires.

What the routers picked

The Inference Router reports the model that served each call in the response's model field, so you can see exactly what it chose:

• router:general sent 571 calls to Qwen3.8-Max and 29 to DeepSeek V4 Flash. It made every call correctly and cost $1.57 per 1,000 alerts, about 5 percent less than calling Qwen3.8-Max directly. For this task, the general router is mostly the expensive model.
• router:software-engineering sent 465 calls to deepseek-v4-flash-0731, 72 to a model it reported as deepseek-4-flash and 63 to Qwen3.8-Max. It cost about $0.31 per 1,000. Its 4 wrong answers all came from the 72 deepseek-4-flash calls: two pages and two ignores, all on ticket-level alerts. The 465 deepseek-v4-flash-0731 calls were all correct, as were all 600 of our direct calls to that model. We do not know what deepseek-4-flash is, and the price page does not list it.

The router is free during the preview, and a routed call costs whatever the chosen model costs. In this test, calling DeepSeek V4 Flash directly was both cheaper and more accurate than the software-engineering router, and the general router cost about the same as calling Qwen3.8-Max directly.

Latency is where the cheap model pays

DeepSeek V4 Flash matched Qwen3.8-Max's accuracy at about a tenth of the token cost. Latency goes the other way. It is a reasoning model, and 81 percent of its output tokens were reasoning, a median of 143 output tokens per call against Qwen3.8-Max's 33 tokens of plain JSON.

Latency per alert: p50 bar, p95 marker

Wall-clock time of the final attempt of each call, 600 calls per model; retries are not included (only Ministral 3 14B needed any, on 11 calls). The models ran one after another with 4 to 6 calls in flight, so service conditions may have differed between them.

DeepSeek V4 Flash had 33 calls over 10 seconds and one at 60.6 seconds. For a ticket, four seconds does not matter. For a page, a minute does. If you put a model in front of your pager, give each call a timeout and decide in advance what a timeout means. The safe default is to page: a model that is slow or down should never be the reason an outage goes unheard.

The bug the models found

In the first run, nobody scored 600. Qwen3.8-Max, gpt-oss-120b and both routers "got wrong" the same two alerts every time. This was one of them:

status: firing
labels: alertname=PostgresPrimaryUnreachable env=production service=postgres-orders severity=critical team=platform
summary: postgres-orders primary not responding
description: The primary of postgres-orders has not answered health checks for 7 minutes.
startsAt: 2026-10-23T05:35:00Z
maintenance window: active until 05:00 UTC, change CHG-4200 (major version upgrade of postgres-orders)

Our label said ignore, because the alert carries a maintenance window. But the window, read as ending at 05:00 that morning, ended before the alert started at 05:35. Our generator picked the start time and the window separately and never checked that one was inside the other. Here is what three models said about this alert in run 1, copied from the run logs:

qwen3.8-max:
{"decision": "page", "reason": "Production primary database unreachable for 7 minutes and the maintenance window ended at 05:00 UTC, so the page criteria apply."}

openai-gpt-oss-120b:
{"decision":"page","reason":"Production primary database is unreachable and no active maintenance window"}

llama-4-maverick:
{"decision": "ignore", "reason": "service is in an active maintenance window"}

Across the two broken alerts and three runs, Qwen3.8-Max, gpt-oss-120b and both routers paged all 6 times. Llama 4 Maverick and Ministral 3 14B trusted the word "active" all 6 times. DeepSeek V4 Flash paged three times, ignored twice and once used up its 2,000-token allowance on reasoning without giving a decision. gpt-oss-20b paged four times and ignored twice. The second reader that "agreed on all 200" had trusted the word too. After the fix, it labelled the two corrected alerts ignore, which matches their labels.

Three lessons from that:

1. Agreement between two readers is not proof the labels are right. Both can take the same shortcut. Here, the disagreements between models found what our check missed, so read them before you trust a score.
2. A score is only as good as its labels. With the bug in place, Llama 4 Maverick ranked first (99.5 percent) and Qwen3.8-Max tied for third (99.0 percent), marked down for reading the timestamps.
3. Real alerts can contain this kind of contradiction. A silence that expired, a maintenance flag that nobody cleared. The models that read the times treated stale context the way an engineer should.

We fixed the inputs, not the labels: the generator now keeps maintenance alerts inside their window, only those two alerts changed (their start times moved to 03:00 and 04:35), and we reran them on every configuration. This correction came after we had seen the results, so treat the final numbers as exploratory rather than a clean holdout. The original two alerts and every configuration's answers to them are in results/v1-maintenance-bug/ in the repo. The window text still has an end time but no date, which a stronger dataset would fix.

What 200 alerts cannot tell you

Perfect scores on 200 alerts do not mean a perfect model. Alerts are the unit here (three runs of one alert are not three independent samples), so we count the alerts a model got wrong in at least one run and compute a one-sided 95 percent upper confidence limit on that share. These limits assume the alerts are a random sample, and they are not: they come from 31 templates. Read them as a sense of scale, not as a bound on production error rates:

So "missed no pages" means "missed none of these 50", which is still consistent with missing about one in seventeen. Compared with Qwen3.8-Max, the other configurations except Ministral 3 14B were between 0 and 0.83 percentage points less accurate. A paired bootstrap over alerts puts router:software-engineering at -0.67 (95% interval -1.33 to -0.17) and gpt-oss-20b at -0.83 (-1.67 to -0.17), while the intervals for gpt-oss-120b and Llama 4 Maverick reach zero. When two models made identical calls, the bootstrap has nothing to resample, and that is not evidence that they are equivalent. We cannot rank DeepSeek V4 Flash, Qwen3.8-Max and router:general against each other from this data.

There is also what this test does not cover:

• The alerts carry their own context. Ours say which service is affected, whether a release is waiting and whether a maintenance window is active. Real alerts often do not, and getting that context into the prompt may be the hard part. We did not test alerts without it.
• One policy, one invented company. A messier policy, or one with judgement calls ("page if it looks bad"), may separate the models more. We did not test that either.
• Only the models our key could call. We did not test the GPT-5 or Claude models.

Cost in perspective

Each call used about 720 input tokens and a median of 20 to 200 output tokens, depending on the model. At 10,000 alerts a month, which is a busy on-call rotation, the estimated token cost is $1.57 for DeepSeek V4 Flash and $16.40 for Qwen3.8-Max. The whole test, 4,800 scored calls plus 48 reruns, came to an estimated $2.63.

That changes the question. Neither bill matters next to a single missed page, so price should be the last thing you choose on. We tested each configuration on all three decisions, not a cascade where cheap rules or a cheap model sort alerts first and only some go to a second model. On this test, DeepSeek V4 Flash and Qwen3.8-Max made the same calls. Qwen3.8-Max answered in 1.3 seconds at the median and never took more than 6.6. DeepSeek V4 Flash cost a tenth as much and sometimes took ten seconds or more. If a few extra seconds before a page are acceptable to you, the cheap model is the better buy. Either way, test it on your own alerts first.

Run it on your own alerts

The generator is the part to replace. Export a few hundred real alerts from Alertmanager, write the decision a senior on-call engineer would make next to each one in the same JSON shape as data/alerts.jsonl, and run the same scripts on a clean results directory. The runner skips any call it already has on disk, so leftover results from our run would be reused:

git clone https://github.com/The-DevOps-Daily/alert-triage-bench
cd alert-triage-bench
cp .env.example .env   # add your model access key
set -a; . ./.env; set +a

rm -rf results/raw && mkdir results/raw                # start clean
node scripts/03-run-models.mjs --runs 3 --budget 5     # stops near $5
node scripts/04-score.mjs                              # writes results/summary.md

The budget is checked after each call, so the calls already in flight can take it a little past the limit.

What we would do with the results:

• Write the policy down first. Every model in this test, even the weak one, beat a deliberately misleading severity label by a wide margin, and all of them were reading the same 31 non-blank lines of policy.
• Score missed pages separately from accuracy. A model can be 99 percent accurate and still miss the one alert that mattered.
• Run each alert more than once. One run hides models that change their mind.
• Read the cases where the models disagree with your labels. In our test, some of them exposed contradictions in our own inputs.
• Time out to a page, not to silence. The model is a filter in front of your pager, never a reason for it to stay quiet.

On this test, with this policy, the cheap model was good enough. The price difference was never the thing to worry about. The latency, the missed pages and the labels were.

The definition alone moved every project across at least two levels. From the same Kubernetes history, deployment frequency is Medium (42 final releases, 0.81 a week), High (63 tags with the alpha, beta and rc tags, 1.21 a week) or Elite (3,286 landings on master, 63 a week). Lead time is High (5.2 days) if a landing on master counts as the deploy, Medium (25.9 days) to the first tag, and Low (76.6 days) to the first final release. For uv, which squash-merges, the lead time from first commit to landing is 0.0 hours, because its squash commits keep no record of when the work started.

TL;DR

• A deploy is a definition. Every project lands in Elite for deployment frequency if each landing on the default branch counts, and in Medium or High if releases count.
• Lead time depends on where the clock starts and stops. Kubernetes ranges from 5.2 days to 81.9 days on the same history, depending on the start timestamp, the event that stops the clock, and whether backports count.
• The merge button can erase history. In 3,090 of uv's 3,132 landings, the author date is within a minute of the commit date. Prometheus uses both buttons: changes landed with a merge commit show 48.2 hours from first commit to landing, single-commit landings (squashed or pushed) show 0.0 hours.
• Change failure rate is the least stable of the four. uv is at 86.1%, 19.8% or 5.0% depending on whether a failure is a quick patch release, any revert, or a revert of a change that shipped.
• What to do: write the definitions down, keep them fixed, and compare a team with its own past, not with a level chart or another tool.

Prerequisites

• git and python3 (standard library only), plus bc for the merge-style demo
• A repository whose tags mark releases with version numbers
• The companion repo dora-from-git, which holds every script and every recorded result in this post

How we measured

We took one year, 1 October 2025 to 1 October 2026, and four projects that merge in different ways:

The scripts use plain Git and nothing else. Go is hosted on Gerrit, not GitHub, and the same code reads it. Treeless clones keep every commit and tag but no file contents, which is enough for metadata: the four repos take 132 MB, and a full run took 137 seconds on a Raspberry Pi 4 (results/benchmark.txt).

The pipeline is four stages, each writing a file the next one reads:

1. 00-clone.sh clones each repo and records its head commit.
2. 01-extract.py records, for every tag in the window, the commits it shipped for the first time, every first-parent commit on the default branch with the commits it brought in, and every revert with the commit or Gerrit change its message names.
3. 02-metrics.py computes each metric under each definition.
4. 03-tiers.py places the results in the 2024 levels.

Before we trusted the extract, check-method.py compared it with plain Git on seeded random samples. The commits each merge brought in matched git rev-list --no-merges <first-parent>..<merge> on 58 of 58 merges, and the first final release that shipped a commit matched git tag --contains on 60 of 60 commits.

The levels come from the 2024 Accelerate State of DevOps report (version 2024.3, page 13):

The report describes these as clusters of survey answers, not as thresholds. We use the ranges the way a dashboard does, as bands. We place only deployment frequency and lead time, because change fail rate is a single value per level and is not in order: Medium (10%) sits below High (20%).

Note

These projects ship releases, not production deploys, so a tag stands in for a deploy. That makes them a fair test of the definitions, and an unfair basis for ranking the projects. Nothing here says one project is better run than another.

What counts as a deploy

Four definitions: every tag, every final release, every day with a final release, and every landing on the default branch (a merge, a squash or a direct commit). A tool that reads only the Git host sees no deploys, so the last one is the event it can count.

Kubernetes shows how much the choice matters. It cut 42 final releases in the year, but only on 15 days, because it patches several release lines on the same day. Count tags, releases or release days and you get three different numbers from the same calendar. Count landings and every project is Elite, including Go, whose master branch is not a release.

Your company has the same choice, with different names: every merge to main, every pipeline run, every rollout to production, or every release train. Pick one and write it down.

Where the lead time clock starts and stops

Lead time has two ends, and Git offers several timestamps for each.

1. First commit author date
2. Last rebase or amend committer date
3. Lands on main merge or submit
4. First tag alpha, beta or rc
5. Final release the deploy

Here is Kubernetes, from the same history, seven ways. The rows do not cover identical sets of changes: a landing-based row counts what landed in the window, a release-based row counts what releases in the window shipped, and each stop event picks up a slightly different set.

Kubernetes change lead time, same history, seven definitions

Median days. 1 Oct 2025 to 1 Oct 2026. Source: results/summary.md in The-DevOps-Daily/dora-from-git.

Four choices do most of the moving:

• Which release stops the clock. Kubernetes cuts alpha tags from master, so a commit reaches its first tag 25.9 days after it is written, and its first final release 76.6 days after. If your release candidates never serve users, they are not deploys.
• Author date or committer date. The committer date changes every time a commit is rebased or amended, the author date does not. On Kubernetes, switching to the committer date moves the median from 76.6 to 63.1 days.
• Backports. For 605 of the 6,091 commits that Kubernetes final releases shipped, we found no landing on master between October 2024 and October 2026. They were made on release branches: mostly cherry-picked fixes, plus the release commits themselves. git cherry-pick keeps the original author date, so a cherry-pick counts the time since the fix was first written. Leaving these commits out moves the median from 76.6 to 81.9 days.
• Per commit or per change. We count a change as one landing on the default branch, so a pull request merged with a merge commit counts once, while a rebased pull request with ten commits counts ten times, and a commit with no landing on master counts on its own, from its commit date. On Kubernetes, per commit and per change barely differ (76.6 against 77.0 days). The unit matters more where pull requests are large.

Go is Low in every release-based definition, from 69.1 days (to the first tag, beta and rc included) to 138.0 days (to the final release, backports left out). It ships a feature release every six months, and in our window go1.26.0 was tagged on 10 February 2026 and go1.27.0 on 19 August 2026. Count a landing on master as the deploy and the same project is High, at 4.1 days.

The other repos show smaller spreads. Prometheus is Low (31.9 to 40.5 days) in every release-based definition. uv releases about twice a week, so it is High (28.6 to 30.1 hours) in all of them.

The merge button decides what Git remembers

This is the part we did not expect to be so clean. With the default behaviour of each button, the way a change lands rewrites its timestamps, and the lead time follows. demo-date-rewrite.sh writes two commits on Monday and Wednesday, lands them on Friday three ways, and reads the result back:

one change, three merge buttons

$ ./scripts/demo-date-rewrite.sh
merge commit:
  feature: part 2 | author 2026-09-09T15:00:00+00:00 | committer 2026-09-09T15:00:00+00:00
  feature: part 1 | author 2026-09-07T10:00:00+00:00 | committer 2026-09-07T10:00:00+00:00
  lead time from author date 102.0 h, from committer date 102.0 h

squash:
  feature (squashed) | author 2026-09-11T16:00:00+00:00 | committer 2026-09-11T16:00:00+00:00
  lead time from author date   0.0 h, from committer date   0.0 h

rebase:
  feature: part 2 | author 2026-09-09T15:00:00+00:00 | committer 2026-09-11T16:00:00+00:00
  feature: part 1 | author 2026-09-07T10:00:00+00:00 | committer 2026-09-11T16:00:00+00:00
  lead time from author date 102.0 h, from committer date   0.0 h

A merge commit keeps the original commits and their dates. A squash writes one new commit dated at the landing. A rebase keeps the author dates but resets the committer dates. Git lets you override both dates, so these are defaults, not laws, and a commit date is not proof of when a commit was pushed.

The real repos behave the same way:

• uv lands every change as a single commit. In 3,090 of its 3,132 landings, the author date is within 60 seconds of the commit date, so "first commit to landing" is 0.0 hours at the median and at the 75th percentile. Git cannot tell how long a uv pull request was open.
• Prometheus uses both buttons. The 626 changes it landed with a merge commit took 48.2 hours from first commit to merge at the median. The 615 squashed or pushed ones show 0.0 hours, and in 612 of them the author date is within a minute of the commit date. Same project, same maintainers, same year.
• Go submits through Gerrit, which rebases but keeps the author date. None of its 3,414 single-commit landings has an author date within 60 seconds of the commit date, and the median from author date to submit is 99.0 hours (4.1 days).

So with these defaults, a team that squash-merges looks Elite on "first commit to landing" lead time however long its reviews take, and a team that switches from merge commits to squash improves that number overnight without changing anything else.

Three rules for a failed deploy

Git has no incidents, so a Git-only tool has to infer failures. We tested three rules a Git-only tool can apply:

Each rule fails in its own way:

• The hotfix rule reads a version scheme as failures. uv increments the patch number for nearly every release and shipped 101 in the year, so the next patch almost always follows within a week: 86.1%. Kubernetes publishes patch releases monthly, so the same rule never fires: 0.0%. Neither number says anything about failures.
• "Ships a revert" counts reverts that users never saw. Many reverts undo a change before it reaches a release. In Kubernetes, 34 reverts that shipped in a final release undid a change that had not shipped in an earlier release in the window: either it was never released, or it shipped before the window.
• "Reverts a shipped change" needs the revert to name its target in a form a script can read. Our parser looks for "reverts commit " and "reverts CL ". It could not resolve 21 Kubernetes reverts and 19 uv reverts: their messages name no commit that way, sometimes none at all, so this rule cannot match them. Go writes "This reverts CL 847405" instead of a commit hash, and the script resolves those through the Reviewed-on trailer that Gerrit adds. Your conventions decide what this rule can see.

The strict rule finds one failure each in Kubernetes, Go and Prometheus. With one event, the rate is noise: one more revert would double it.

Recovery time inherits all of this. Under the hotfix rule it is the time to the next patch: 3.0 days for uv, 5.2 for Prometheus, 6.0 for Go. It can never be longer than 7 days, because the rule only counts a failure when the fix came within 7 days. The number measures the rule, not the recovery.

What a dashboard adds, and what it does not

The script shows what Git alone supports. A full engineering-metrics tool (vendors in this space include LinearB, Swarmia, Jellyfish, DX, Sleuth and Faros AI) can read other systems, and those systems hold data Git does not:

• When the work started. The forge API knows when a pull request was opened and when its first commit was pushed, which is exactly what a squash merge removes from Git.
• Real deploy events. A CD pipeline knows when a version reached production, per environment. A tag is a stand-in.
• Real failures. Linking deploys to incidents, rollbacks and pages measures failure directly instead of inferring it from reverts or patch numbers.
• Ownership and trends. Mapping repos and paths to services and teams, and recomputing the same way every week.

What a tool does not remove is the choice of definition. Before you compare a number across teams or tools, ask which event counts as a deploy, which timestamp starts lead time, whether backports and release candidates count, and what counts as a failure. If the answers differ, the numbers do not compare.

What we could not conclude

• These projects do not deploy. Tags stand in for deploys. The results test the definitions, not the projects.
• The levels are survey clusters. The report presents them as groups of answers, and the band edges are our reading.
• The failure numbers are small. The strict revert rule found one failure in three of the four projects, so its rates move a lot with one release.
• Unshipped work is left out. Lead time counts only commits that reached a release before the window closed, so the medians describe completed work. Go's work since the go1.27 freeze is not in them.
• Tag dates differ. uv and Go use lightweight tags, dated by the tagged commit. Kubernetes and Prometheus use annotated tags, dated by the tagger. A release can be published hours after its tag.

Run it on your repositories

git clone https://github.com/The-DevOps-Daily/dora-from-git.git
cd dora-from-git
./run.sh
cat results/summary.md

To measure your own repositories, edit repos.json. Each entry needs a clone URL, the default branch, and a regular expression for your release tags, with groups for major, minor and patch and, if you tag release candidates, a fourth group for the prerelease suffix:

{
  "name": "payments-api",
  "url": "git@gitlab.example.com:platform/payments-api.git",
  "branch": "main",
  "merge_style": "squash",
  "tag_pattern": "^v(\\d+)\\.(\\d+)\\.(\\d+)(-rc\\.\\d+)?$"
}

If your deploys are not tagged, have your pipeline tag each production deploy with a version that the pattern matches, for example v1.42.0, and the same scripts measure deploys instead of releases. The hotfix rule reads the patch number, so it only means something if a patch bump marks a fix.

The-DevOps-Daily/dora-from-git on GitHub

Summary

• The same Git history put each of the four projects in two or three different DORA levels, depending only on the definition.
• What counts as a deploy moved deployment frequency by up to two levels, and every project is Elite if a landing on the default branch counts.
• Where the clock starts and stops moved Kubernetes lead time from 5.2 to 81.9 days on the same history.
• With default settings, squash merges and rebases rewrite the timestamps lead time depends on, so the merge button can change the metric without changing the work.
• Change failure rate inferred from Git measured version schemes and commit message habits more than failures.
• Write the definitions down, keep them fixed, record the unit (commit or change), and compare a team with its own past.

Every "Datadog alternatives" list says the alternatives are cheaper. Few say by how much, because the vendors price in different units: per GB, per event, per host, per user, per compute credit. So we did the arithmetic. On October 12, 2026 we read the pricing page of Datadog and 12 alternatives, and priced one workload on each at list price: 300 GB of logs a month, kept for 30 days. Then we wrote down what each tool is good at and the catch we found on its pricing page.

TLDR

• The workload: 300 GB of logs a month, 30-day retention, about 1 KB per log event (so 300 million events), list prices, no negotiated discounts.
• Datadog: $780 a month with annual billing if you index every event with 30-day retention, $1,155 on demand. Index 10% of events and it drops to $105, but the other 90% are then not searchable unless you archive them or use Flex Logs.
• Cheapest on paper: Uptrace ($19 to $25, its page gives two rates), Axiom ($25 if your logs compress at least 3:1 and you stay inside the included query compute), Better Stack ($45 in its EU region).
• Generous free tiers: Axiom Personal (500 GB of data loading), New Relic (100 GB of ingest), Grafana Cloud (50 GB each of logs, traces and profiles).
• Best fit by need: Grafana Cloud if you already run Prometheus, Honeycomb for debugging traces, SigNoz or OpenObserve if you may self-host later, ClickStack if you want ClickHouse and long retention.
• The catches: a 30% annual discount folded into a "pay as you go" price, ingest that stops when you pass the free tier, a daily quota that blocks data until midnight UTC, and region-based prices that differ 3.5x.

Prerequisites

• A rough idea of your monthly log volume in GB. Your current vendor's usage page has it.
• Your average log event size. If you do not know it, divide bytes ingested by events ingested on your usage page.
• Ten minutes with the OpenTelemetry Collector docs if you plan to try two tools side by side (more on that at the end).

How we priced it

Prices change often, so everything below is a snapshot of each vendor's own pricing page on October 12, 2026. We did not use third-party price lists.

The workload is logs only, because logs are the line that sends most people looking, and because every vendor prices logs in a way we can convert:

• Volume: 300 GB ingested per month (about 10 GB a day).
• Retention: 30 days. Where the listed price includes less, we say so.
• Event size: 1 KB on average, so 1 GB is 1 million events and the month is 300 million events. This matters for the vendors that bill per event. If your events are smaller, per-event pricing gets worse fast. Our logs cost post measured 209 bytes per nginx line on a real server, which is about 5 million events per GB.
• Not included: hosts, APM, metrics, support plans, cloud egress from wherever your workloads run, and query charges where they are separate (OpenObserve and Axiom bill queries; most others include them). We add a required user or plan fee where the vendor needs one to send logs at all, and call out the rest per vendor.

Monthly list price for 300 GB of logs

Vendor pricing pages, October 12, 2026. 1 KB per event (300M events), 30-day retention unless the label says otherwise. Logs only; query charges extra on OpenObserve and Axiom. Uptrace shows the higher of the two rates on its page. Axiom assumes at least 3:1 compression. ClickStack, Coralogix, Elastic, Honeycomb and self-hosting are left out because their pages do not give a single comparable figure; see each section.

A cheaper log line does not make a cheaper platform. Most of these tools also charge for metrics, traces or seats, and some charge for queries. Use the number to see the gap, then price your own mix with each vendor's calculator.

The baseline: Datadog

Pricing model: per host for infrastructure and APM, per GB to ingest logs, per million events to index them. Infrastructure Pro is $15 per host per month billed annually ($18 on demand). APM is $31 per host ($36 on demand).

300 GB of logs: ingest is $0.10 per GB, so $30. Indexing with 30-day retention is $2.50 per million events billed annually, $3 billed monthly and $3.75 on demand. At 300 million events:

annual:     $30 ingest + 300 x $2.50 = $780 per month
on demand:  $30 ingest + 300 x $3.75 = $1,155 per month
index 10%:  $30 ingest +  30 x $2.50 = $105 per month

Fair to Datadog: you rarely need to index every log. Exclusion filters, shorter retention (7-day indexing is $1.27 per million) and Flex Logs (storage at $0.05 per million events per month, with compute for queries priced separately) all cut the bill. The 10% figure above is only fair if the other 90% can stay unsearchable or go to an archive. Datadog also lists more than 1,000 integrations.

The catch: the billable host count is the "high-water mark of the lower 99 percent" of hourly usage. The top 1% of hours (about 7 hours in a 730-hour month) are dropped, so an autoscaling spike that lasts longer than that sets your host bill for the month. On the log side, a spike raises the ingest charge, and the indexing charge too if those events pass your index filters.

1. Grafana Cloud

Best for: teams that already run Prometheus, Loki or Grafana dashboards.

Pricing model: Pro is "from $19/month + usage". Logs, traces and profiles start at $0.05 per GB to process plus $0.40 per GB to write, and the first 30 days of retention are included. Metrics start at $6.50 per 1,000 active series.

Free tier: 10,000 active series, 50 GB each of logs, traces and profiles, 3 active users, 14-day retention.

300 GB of logs: $19 + (300 - 50) x $0.45 = $131.50.

Strengths: the open source stack underneath (Grafana, Loki, Tempo and Mimir, all AGPL-3.0), native PromQL, and a free tier that covers logs, traces and profiles. You can move to self-hosting later without changing query languages.

The catch: Loki indexes labels, not log content. Its own docs warn that too many label combinations give "a large index and small chunks", which "can actually reduce performance". Plan your labels before you send 300 GB. The pricing page also carries a banner saying Grafana is changing its pricing, so check it again before you commit.

2. SigNoz

Best for: teams that want one per-GB rate for logs and traces, and the option to self-host.

Pricing model: the Teams plan is $49 a month, and that includes $49 of usage. After that, logs and traces are $0.30 per GB ingested and metrics are $0.10 per million samples. No per-host or per-user fees.

Free tier: a cloud trial. Self-hosting is free; the code is MIT licensed apart from the ee/ and cmd/enterprise/ directories.

300 GB of logs: 300 x $0.30 = $90, which absorbs the $49 minimum, so $90, at the listed 15-day retention. The page offers 30, 90 and 180 days and a year, but the text we captured does not give their prices.

Strengths: built on OpenTelemetry and ClickHouse, with OTLP over gRPC and HTTP. Logs, traces and metrics share one UI and one query builder.

The catch: SSO and SAML are cloud only, RBAC is in beta, and audit logs and multi-tenancy are listed as "coming soon". If compliance features matter, check where they stand.

3. Better Stack

Best for: small teams that want uptime checks, on-call, status pages and logs on one bill.

Pricing model: per GB ingested plus per GB-month retained, priced by region. In Germany (the default) ingest is $0.10 per GB and retention $0.05 per GB per month. In the US it is $0.15 and $0.08. In Singapore $0.35 and $0.18. On-call responders are $29 a month per seat billed annually.

Free tier: 10 monitors, 1 status page, 3 GB of logs and 3 GB of traces (kept for 3 days), 30 GB of metrics.

300 GB of logs: 300 x $0.10 + 300 x $0.05 = $45 in Germany, $69 in the US. The plan includes 3 GB a month (kept for 3 days), which changes this by less than a dollar.

Strengths: one of the lowest per-GB list prices here, a full UI, and OTLP over HTTP. You can keep the data in your own S3 bucket for $250 a month.

The catch: region matters more than you would guess: Singapore ingest costs 3.5x the German rate. The pricing page also says it is "currently not HIPAA compliant".

4. ClickStack (HyperDX on ClickHouse Cloud)

Best for: teams that want long retention on cheap storage, or already use ClickHouse.

Pricing model: you pay for ClickHouse Cloud compute and storage, not per GB of logs. Storage is $25.30 per TB per month. Compute is metered per minute at $0.2181 per unit-hour on Basic (a unit is 8 GiB of RAM and 2 vCPUs, AWS us-east-1), and the Basic tier offers 8 to 12 GiB of memory. Compute scales to zero when idle.

Free tier: $300 of credits for 30 days. HyperDX is MIT and ClickHouse is Apache-2.0, so you can self-host both.

300 GB of logs: it depends on how long compute runs. Logs arrive all day, so assume one 8 GiB unit stays awake: $0.2181 x 730 hours = $159.21. Storage at ClickHouse's own 10x compression estimate is 30 GB, about $0.76. That gives about $160, before backups and query load. We left it off the chart because the real figure depends on sizing you choose.

Strengths: storage is so cheap that a year of retention changes the bill very little. SQL on your logs, and native OTLP.

The catch: you are sizing a database, not buying a per-GB service. Continuous ingestion keeps compute awake, so at small volume the compute line dominates the bill.

5. OpenObserve

Best for: teams that want a cheap self-hosted option with a cloud fallback.

Pricing model: $0.50 per GB ingested plus $0.01 per GB queried, with 30 days of log retention included.

Free tier: a 14-day cloud trial. The self-hosted Enterprise edition is free up to 50 GB a day, and the open source version is AGPL-3.0.

300 GB of logs: 300 x $0.50 = $150 at the annual-commitment rate (see the catch), plus queries (scanning 1 TB in a month adds $10).

Strengths: the free self-hosted tier is generous: 50 GB a day is 1.5 TB a month, five times our workload.

The catch: the page calls the $0.50 rate "pay-as-you-go, billed monthly", but a footnote says it "includes 30% discount for annual commitment". The page does not show the price without the commitment.

6. Axiom

Best for: high log volume with modest query load.

Pricing model: the Axiom Cloud plan is $25 a month, and that includes 1,000 GB of data loading, 100 GB-hours of query compute and 100 GB of storage. Past that, data loading starts at $0.12 per GB, queries at $0.20 per GB-hour, and storage at $0.03 per GB per month, charged on compressed data.

Free tier: the Personal plan gives 500 GB of data loading, 10 GB-hours of queries, 25 GB of storage and 30-day retention, with no card.

300 GB of logs: the loading fits in the included 1,000 GB. Storage is charged on compressed data, so 300 GB fits in the included 100 GB at any compression ratio better than 3:1. That leaves $25, plus query compute if you go past 100 GB-hours. If your logs compress worse than 3:1, storage adds up to $6 (at no compression, 200 GB over the allowance at $0.03).

Strengths: one of the cheapest results for our workload, and the largest free allowance for data loading on this list.

The catch: queries are metered, and query compute from AI agents counts the same as from people. A team that queries all day, or wires an assistant into its logs, should model query cost first.

7. Honeycomb

Best for: debugging distributed systems through traces with many unique attribute values.

Pricing model: per event, with unlimited seats. Each span, span event and link counts as an event. Pro starts at $150 a month for 50 million events and goes up to 750 million.

Free tier: 20 million events and 100 million metric data points a month.

300 GB of logs: our 300 million events fit inside Pro's range, but the page does not list the price for that tier, so we left it off the chart.

Strengths: built for asking new questions of trace data. High-cardinality fields (user ID, build ID, feature flag) cost nothing extra, because pricing is per event, not per field.

The catch: 60 days of retention are included, and longer retention is an Enterprise add-on. For our workload you need a quote, and per-event pricing means smaller events cost more per GB.

8. New Relic

Best for: small teams that fit in the free tier, and teams that want no host-based pricing.

Pricing model: $0.40 per GB beyond 100 free GB a month, plus users. Core users are $49 a month. A full platform user on Standard is $10 for the first and $99 each after that (up to 5).

Free tier: 100 GB of ingest a month, 1 full user, unlimited basic users.

300 GB of logs: (300 - 100) x $0.40 = $80, plus $10 for the first full platform user on Standard, so $90. The pricing page we read does not state log retention on paid plans (the free tier keeps data for at least 8 days), so confirm it covers 30 days.

Strengths: 100 GB of free ingest a month, OTLP support and no per-host charges.

The catch: on the free tier, going over 100 GB means "data ingest will cease and you will lose access to the platform" until you upgrade to a paid edition or the next month starts. New Relic says it warns you first, but it is still a bad surprise during an incident, which is exactly when log volume spikes.

9. Dash0

Best for: teams already standardized on OpenTelemetry.

Pricing model: per million signals, with separate ingest and storage prices. Spans and log records cost $0.060 per million to ingest and $0.540 per million to store for 30 days.

Free tier: a 14-day trial.

300 GB of logs: 300 x ($0.060 + $0.540) = $180.

Strengths: built around OpenTelemetry from the start, and storage is billed only on what you keep.

The catch: data you filter out still pays the ingest price, and per-event pricing punishes small events. At the 209-byte nginx line from our logs post, the same 300 GB would be about 1.4 billion events.

10. Coralogix

Best for: teams that want to keep logs in their own bucket and still query them.

Pricing model: units at $1.50 each, charged per GB by pipeline against a daily quota. The headline prices are $0.42 per GB for logs, $0.16 for traces and $0.06 for metrics.

Free tier: a 14-day trial with 8 units.

300 GB of logs: $126 to $345, depending on the pipeline. The headline $0.42 per GB gives $126. The FAQ on the same page prices Monitoring at $0.50 per GB ($150) and Frequent Search at $1.15 per GB ($345). The page does not say which pipeline the $0.42 applies to, which is why it is not on the chart.

Strengths: you can query the archive in your own bucket, and the priority pipelines let you pay less for logs you only keep for compliance.

The catch: when you hit the daily quota (2x with pay-as-you-go turned on), data is blocked until 00:00 UTC, and unused units expire.

11. Uptrace

Best for: cost-sensitive teams in Europe.

Pricing model: per GB for spans and logs, per million data points for metrics, 28-day retention. The page says "50 GB free/mo · then $0.075/GB" in one place and rates that "start at $0.10/GB" in another.

Free tier: 50 GB and 5,000 time series a month. The self-hosted Community Edition is AGPL-3.0.

300 GB of logs: (300 - 50) x $0.075 = $18.75, or $25 at $0.10. Call it $19 to $25, and check the page before you budget.

Strengths: among the lowest per-GB rates on this list, with unlimited users and hosts.

The catch: the pricing page contradicts itself on the starting rate, and the cloud runs in one region, Hetzner in Germany. That suits EU data residency, but workloads in the US or Asia send every log across an ocean.

12. Elastic Observability (Serverless)

Best for: teams that need full-text search across log content.

Pricing model: per GB ingested plus per GB retained. Logs Essentials is "as low as $0.07/GB" to ingest and "as low as" $0.017 to retain. The Complete tier lists logs and traces "as low as $0.09/GB". Limited support is included; higher support levels add 5% to 15%.

Free tier: a 14-day trial.

300 GB of logs: not computed. The page shows only the lowest volume tier ("as low as"), not the rate at 300 GB.

Strengths: it indexes the text of every log line, so full-text searches over long time ranges stay fast. Loki can also search log content, but it scans the chunks instead of using an index. Self-managed and hosted options exist.

The catch: the page shows only "as low as" rates, so you need the calculator for a real number.

The self-hosted option: Prometheus, Grafana, Loki and Tempo

Best for: teams with spare infrastructure and the people to run it.

Pricing model: no license fee. Prometheus is Apache-2.0. Grafana, Loki, Tempo and Mimir are AGPL-3.0.

300 GB of logs: whatever your servers, object storage and on-call time cost. For 300 GB a month, Loki on object storage is cheap to store. The real cost is the people.

Strengths: full control, no per-GB pricing, and every piece speaks OpenTelemetry: Prometheus has an OTLP receiver (off by default, --web.enable-otlp-receiver), Loki accepts OTLP at /otlp, and Tempo takes OTLP natively.

The catch: four systems to deploy, upgrade, scale and back up. If nobody owns them, they rot.

How to switch without a big-bang migration

Do not move everything at once. The cheap, safe way is to send the same data to two places and compare:

1. Put an OpenTelemetry Collector in front of your apps if you do not have one. Our post on what to retire now that OpenTelemetry has graduated covers the move.
2. Add a second exporter that sends logs to the tool you want to test. The Collector can fan out to several backends from one pipeline:

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
  otlphttp/candidate:
    endpoint: https://otlp.example-vendor.com   # the candidate's OTLP endpoint
    headers:
      authorization: Bearer ${env:CANDIDATE_TOKEN}

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog, otlphttp/candidate]   # send to both while you compare

1. Run both for a full billing cycle. One week is not enough, because month-end batch jobs and incident spikes change the volume.
2. Compare three things: the bill, whether your top 10 saved searches and alerts still work, and how long a real incident investigation takes in each tool.
3. Move logs first. Logs are often the largest and most portable line. Keep APM where it is until the logs move is done.

Summary

• At list price, 300 GB of logs a month costs $780 on Datadog with annual billing if you index every event, and about $19 to $180 on the alternatives we could price. The biggest saving on Datadog itself is to index less.
• The cheapest log line is not the cheapest platform. Axiom, Uptrace and Better Stack win on logs alone. Grafana Cloud and New Relic win on free tiers. Honeycomb wins on trace debugging. ClickStack wins on long retention.
• Read the footnotes. We found a 30% annual discount in a "pay as you go" price, ingest that stops at the free-tier limit, a daily quota that blocks data until midnight UTC, regional prices 3.5x apart, and a page that gives two different rates.
• Event size changes everything for per-event vendors. Measure yours before you trust any comparison, including this one.
• Prices change. Every number here is from the vendors' own pages on October 12, 2026.

You cannot memorize your way through that, but you can practice it. This post covers what interviewers grade in a situational question, a five-step way to structure your answer, five worked scenarios with a weak and a stronger answer for each, and a practice routine that trains the decision instead of the definition.

TLDR

• Situational questions grade your reasoning, not your recall. The same final action can score well or badly depending on how you got there.
• Structure every answer the same way: scope the problem, name your first check, pick the reversible move, say what you tell people, and say what you change afterwards.
• Always say the risk you accept out loud. "I roll back, and we lose the new feature for the evening" is a stronger answer than "I roll back."
• Practice by changing one fact and answering again. The decision changes when the migration is destructive, when it is 3 a.m., or when the VP says no. That boundary is what interviewers probe.

Prerequisites

• Some hands-on time with production systems. The scenarios assume you know roughly what a deploy pipeline, a Postgres primary, a Terraform plan and an AWS access key are.
• A timer and a way to record yourself. Speaking an answer out loud is a different skill from thinking it.

What interviewers are actually grading

A situational question is a small simulation of the job. The interviewer has a rough rubric in their head, even if nobody wrote it down. Here is a version of it, with what a weak and a strong signal sound like:

Notice that none of the rows is "knows the right tool". Tools come up, but as evidence for a decision, not as the answer. A candidate who says "I'd run kubectl rollout undo" and stops has given a command. A candidate who says "the errors started with the rollout and only on the new pods, so I undo the rollout first, because it takes a minute and can be redone" has given a decision.

A structure that works for any scenario

You do not need a framework with a clever acronym. You need the same five moves every time, so the interviewer can follow you and you do not forget the communication part under pressure.

Goal: Say the risk you accept out loud

1. Scope who, what, since when
2. First check one signal, and why
3. Decide reversible move first
4. Communicate who hears what, next update
5. Follow up fix the class, not the case

new evidence changes the decision, then back to step 1.

1. Scope. Restate the problem with the facts you have, and ask for the ones you need. "How many users? Is it one region? Did anything change in the last hour?" Asking is not stalling. It is the first thing a good on-call engineer does.
2. First check. Name one specific thing you look at first and what each result would mean. Good first checks split the problem in half: "if the spike starts with the rollout, the deploy is the main suspect; if it started before, I look elsewhere first."
3. Decide. Pick the move that reduces harm fastest and can be undone. Rollbacks, feature flags, scaling up and failing over are usually reversible. Deleting data, dropping replication slots and force-pushing are not. If you choose an irreversible step, say why the alternative is worse.
4. Communicate. Say who you tell, what you say, and when the next update is. Write the actual message if you can. Interviewers notice when candidates forget this, because in real incidents the silence is often what makes it worse.
5. Follow up. Say what you change so this does not happen again: a test, an alert, a guardrail in the pipeline, a runbook. This is usually where senior answers separate from junior ones.

Tip

When new information arrives, say so and go back through the loop: "If the migration turns out to drop a column, the rollback is no longer safe, so I would switch to fixing forward." Interviewers often add a twist on purpose to see whether you update your decision or defend your first one.

Five scenarios, worked through

Each scenario below has a short weak answer and a stronger one. The stronger answers are not the only right answers. They are examples of reasoning that would score well, and the point of each is the decision and its trade-off, not the specific commands.

1. The Friday deploy

You deployed checkout v2.31 at 16:40. By 16:50 the 5xx rate on checkout is 2.5%, up from 0.2%. The release also ran a database migration. What do you do?

Weak answer: "I'd look at the logs, find the bug and fix it." Or: "Roll back immediately."

A stronger answer:

1. Scope: Is it all checkout requests or one endpoint? Did the error rate jump at the moment the new pods went live? Are payments failing, or only something cosmetic?
2. First check: Compare the error graph with the rollout timeline and split errors by version. If the old pods are clean and the new ones fail on the same traffic, the release is the main suspect.
3. Decide: The key question is the migration. If it only added a nullable column or a new table, and the old code works with the new schema and with any rows the new code has already written, I roll back the application and leave the migration in place. If the migration dropped or renamed a column the old code needs, a rollback makes things worse, so I fix forward, or turn the feature off behind its flag if the flag really avoids the broken path.
4. Risk I accept: Rolling back loses the new feature until Monday and may annoy the team that wanted it out. That is cheaper than a weekend of failed checkouts.
5. Communicate: In the incident channel: "Checkout errors at 2.5% since the 16:40 deploy of v2.31. Rolling back the app now; the migration stays because it is additive. Next update 17:15." Then tell support, because they will get the customer emails first.
6. Follow up: Why did the canary or the staging run not catch it? Should migrations follow an expand-and-contract sequence, so the previous release always works with the current schema?

2. The database disk is almost full

The primary Postgres instance is at 95% disk and growing by about 1% every ten minutes. What do you do?

Weak answer: "I'd delete old files to free up space."

A stronger answer:

1. Scope: At this rate there are about 50 minutes left. What happens at 100%? Writes fail, and Postgres may shut down. That makes this urgent but not a reason to act blindly.
2. First check: What is growing? The usual suspects are WAL files held back by a replication slot whose replica is down, a long-running query writing temp files, a runaway table or log, or a bulk import. pg_replication_slots and the size of the WAL directory answer the first question quickly.
3. Decide: Buy time with a safe move first. On many managed databases and cloud disks you can grow storage online; on a self-managed disk you also have to extend the filesystem. Check that the resize will finish before the disk fills. Growing is one-way (you usually cannot shrink it back), but it does not risk the data. I do not delete files I do not understand: deleting WAL by hand can corrupt the database.
4. The trade-off they are fishing for: If an inactive replication slot is holding the WAL, dropping it lets Postgres remove that WAL at the next checkpoints. But whatever used the slot (a replica, or a logical replication or CDC consumer) may no longer be able to catch up and may need a rebuild or a resync. That is a real decision: I find out who owns the slot, drop it only if the disk cannot grow in time, and say so out loud.
5. Communicate: Tell the database owner and the teams whose writes would fail, with the time left and what I am doing. Update when space is secured.
6. Follow up: An alert on disk growth rate, not only on percentage. An alert on replication slot lag. A runbook for this exact case.

3. The Terraform plan that replaces production

The release pipeline is blocked on a Terraform plan. It shows -/+ destroy and then create replacement for the production RDS instance. The release manager asks you to approve it so the release can go out.

Weak answer: "The plan is what the code says, so I'd approve it."

A stronger answer:

1. Scope: A -/+ replacement destroys the existing database before it creates the new one. Unless a restore and a cutover are planned, that means downtime and probably data loss. That is not a routine change, whatever the pipeline says.
2. First check: Read which attribute the plan marks with # forces replacement. Some attributes cannot change in place, so changing them forces a new instance. I also check the rest of the plan: if someone renamed the resource or moved it into a module, Terraform shows a destroy at the old address and a create at the new one instead.
3. Decide: Do not approve. If it is a rename or a refactor, the fix is a moved block (or terraform state mv) so Terraform knows it is the same resource. If the attribute really cannot change in place, that becomes a planned migration with a snapshot, a restore and a maintenance window, not a pipeline click.
4. Risk I accept: The release is late, and the release manager is unhappy. I can live with that. I cannot undo a destroyed production database.
5. Communicate: Tell the release manager directly: "The plan would replace the production database because of X. I am blocking it; the fix is a moved block, and I expect to have a clean plan in an hour."
6. Follow up: Add lifecycle { prevent_destroy = true } to stateful resources. It only protects a resource while its configuration exists, so also make the pipeline fail on any plan that deletes or replaces a resource tagged as stateful.

4. A leaked cloud key

A developer pushed an AWS access key to a public GitHub repository 20 minutes ago. What do you do?

Weak answer: "Remove the commit from the history and force-push."

A stronger answer:

1. Scope: Public repositories are scanned for keys within minutes, so assume it has already been copied, and treat it as a security incident from the start. Removing the commit does not help with that.
2. First check: Which identity does the key belong to, and what can it do? I start a CloudTrail search for its use across accounts and regions, knowing that events arrive with a delay and that a clean result does not prove the key was not used.
3. Decide: Deactivate the key now, even though something that depends on it will break. This is the case where the disruptive move is the right one, because the alternative is an attacker with unknown access for an unknown time. Then I check whether anyone used it to create other credentials, roles or temporary sessions, because deactivating the key does not cancel temporary credentials already issued with it.
4. Risk I accept: The service using that key goes down until it has a new credential. I say that before I do it, not after.
5. Communicate: Tell security and the owner of the affected service right away, and follow the security incident process.
6. Follow up: Secret scanning on push, short-lived credentials (for example OIDC federation from CI instead of stored keys), and a check of what else that identity could reach.

5. Pressure to skip a check

A release is promised to a customer today. The security scan in the pipeline fails on a dependency, and the release manager asks you to disable the scan so the release can ship.

Weak answer: "I'd disable it, ship, and fix it next sprint." Or: "Absolutely not, security comes first."

A stronger answer:

1. Scope: What exactly is failing? A known exploitable vulnerability in code we actually call is very different from a low-severity finding in a test-only dependency, or a flaky scanner.
2. First check: Read the finding: severity, whether the vulnerable path is reachable, and whether a patched version exists.
3. Decide: Do not silently turn the scan off. If the finding is real and reachable, try a patched version first; if there is none, ship without the affected feature, but only if that really removes the vulnerable code from the build. If it is low risk or a false positive, use a documented, time-limited exception for that one finding, with an owner and an expiry date, instead of disabling the whole check.
4. Risk I accept: With an exception, we knowingly carry one finding for a few days. That is acceptable only if the risk is assessed, someone with the authority to accept security risk approves it, and it has an owner and an expiry date.
5. Communicate: Put the decision and who approved it in writing, in the ticket or the release notes. Interviewers listen for this, because unrecorded exceptions are how temporary bypasses become permanent.
6. Follow up: Review why a release promise was made without time for the pipeline to pass, and make exceptions expire automatically.

How to practice making the call

Reading good answers helps a little. Saying your own answers out loud, under a time limit, helps a lot more. A routine that works:

1. Build a deck of scenarios. Write 20 short prompts from your own incidents, from public postmortems, and from the questions above. Keep each one to three or four sentences, like a real interviewer would.
2. Answer out loud in three minutes. Use a timer and record yourself. Three minutes forces you to pick a first check instead of listing ten.
3. Score yourself with the rubric table. Did you scope? Did you name one first check and what it rules out? Did you say the risk you accept? Did you write the update message? Most people find they skip communication and follow-up.
4. Change one fact and answer again. This is the most useful step. Make the migration destructive. Make it 3 a.m. and you are alone. Make the VP say "do not roll back". If your answer does not change when the facts change, you are reciting, not deciding.
5. Practice with someone who pushes back. A friend or colleague who interrupts with "the logs show nothing, now what?" is closer to a real interview than any list of questions.
6. Write the message, literally. For each scenario, write the two-line update you would post in the incident channel. It takes thirty seconds and it is the part most candidates forget.

You can also practice parts of this in tools that put you under time pressure. The incident commander simulator makes you triage and decide while the impact grows. The interview questions practice mode has scenario-style questions on topics like incident postmortems and progressive delivery rollbacks, and the on-call rotations flashcards cover the vocabulary. If you want feedback on full written answers, the mock interviews on DevOps Daily Pro ask a follow-up question about your answer, the way a real interviewer would.

Note

If you are the one running interviews, the same structure works in reverse. Give a short scenario, let the candidate scope it, then change one fact halfway through. The way they update their decision tells you more than any definition question.

Summary

Situational questions are not a trick. They are a way to watch you work without a real outage. The interviewer wants to see you scope the problem, pick a first check that splits it in half, choose the move you can undo, say the cost of that choice, tell the right people, and fix the class of failure afterwards. You get better at that the same way you get better at incidents: by making the call many times, with the facts changing each time. Build a small deck, answer out loud against a timer, change one fact and answer again. Practice until you use the structure without thinking about it, so in the interview you can think about the problem instead of about how to sound senior.

The engineering blogs explain the architecture. What they rarely show is what each piece costs. So we built the core primitives ourselves, a hash vindex, a router, a lookup vindex and an online reshard, over five real MySQL 8.4 servers, loaded 200,000 chat messages and measured. The router is about 170 lines of Python, so you can read every decision.

TLDR

• The sharding key decides the price of a query. Fetching one message by id with its channel known took 1 shard query, and our setup served 1,194 of those reads per second. Without the channel, the router had to ask all 4 shards: 307 reads per second, about a quarter.
• A lookup vindex buys part of it back: 2 shard queries per read and 710 reads per second, 2.3x the scatter version and 59% of the targeted one, at the cost of an extra write on every insert.
• Cross-shard transactions without atomic commit can leave partial results. We simulated a failure between the two commits in 52 of 2,000 transfers between accounts on different shards, and all 52 lost their 10 credits. The same failure on single-shard transfers lost nothing.
• An online split lost nothing. We split a shard in two, all three tables, while a writer kept inserting and a reader kept querying: 0 write errors, every one of 2,158 lookups found its message, 2,158 scatter reads returned no duplicates, and at the end every expected message id existed exactly once, on the right shard. Insert latency roughly doubled while the copy ran.
• More shards do not fix a hot key. The busiest channel held 12.3% of all messages. At 128 shards, where the ideal is 0.78% per shard, the busiest shard still held 14.6%.

Prerequisites

• Familiarity with MySQL and with what replication and a primary are.
• To run the demo: Linux with sudo, curl and about 3 GB of disk. It builds an Ubuntu chroot with MySQL inside, so nothing is installed on your host. The setup downloads the arm64 image (we used a Raspberry Pi 4); change IMG in scripts/00-setup.sh on x86.

The-DevOps-Daily/mysql-shard-router on GitHub

Why these companies shard MySQL at all

Read replicas fix read load. Once writes or the dataset outgrow the biggest primary you can buy, after you have tuned what you can, horizontal sharding is what is left. The Vitess history page puts YouTube's problem in that order: replicas came first, then "write traffic became too high for the primary database to handle, requiring YouTube to shard data." Connections were the other pressure. The Vitess docs put each MySQL connection at between 256 KB and almost 3 MB of memory, and Vitess's answer was to multiplex many light client connections onto a small pool of real ones.

Resharding was the painful part. Sugu Sougoumarane, one of Vitess's creators, told Percona about YouTube's first one: "It took an entire night of master downtime."

Slack's reason was a bad sharding key. Its original design put each workspace on one shard with all of its data. As big customers grew, their shards "reached the largest available hardware", while the fleet was "hitting some major hotspots, while also massively underutilizing the majority" of the databases. Slack re-sharded messages by channel id, migrating from the fall of 2016 to the end of 2020 with double writes and a parallel read-diffing system to compare old and new.

GitHub did the cheaper thing first. In 2021 it described splitting its main mysql1 cluster by schema domain, groups of tables that are queried together, with linters that stopped a query from joining tables across domains. It moved its 130 busiest tables at once. In 2019 mysql1 answered 950,000 queries per second; by 2021 the split clusters answered 1.2 million and the average load per host had halved. By 2023 GitHub ran 1,200+ MySQL hosts, 300+ TB and 5.5 million queries per second, with horizontally sharded Vitess clusters for the large domains that outgrew a single primary.

The model: keyspace IDs, vindexes and ranges

Vitess does not hash a row straight to a server. It maps the sharding key to a keyspace ID (with a 64-bit hash vindex, as in our demo, that is 8 bytes), and each shard owns a range of that ID space. The function that does the mapping is a vindex. The primary vindex is to sharding what the primary key is to a table: every sharded table needs one.

Shard names are the ranges, in hex: -40 owns IDs from 0x00 up to but not including 0x40, and c0- owns 0xc0 to the end. Four equal shards are -40, 40-80, 80-c0 and c0-. Splitting c0- gives c0-e0 and e0-, and no row outside that range has to move. That is the property that makes resharding possible without rehashing everything.

1. channel_id 1 sharding key
2. hash vindex 8-byte keyspace ID
3. 0xf6fc42… falls in c0-
4. shard c0- MySQL :3314

What we built

Five MySQL 8.4 servers, one per shard plus one spare for the reshard, all on one Raspberry Pi 4 inside an Ubuntu chroot. A Python router plays the part of VTGate: it hashes the key, picks the shard, and for queries without a key, sends the query to every shard in parallel and merges the results.

1. client threads
2. router vindex + shard map

• Raspberry Pi 4 Ubuntu 26.04 chroot
  • -40 mysqld :3311
  • 40-80 mysqld :3312
  • 80-c0 mysqld :3313
  • c0- mysqld :3314
  • spare mysqld :3315

The data is a chat app, like Slack's messages keyspace: 200,000 messages in 2,000 channels, sharded by channel_id. Channel activity follows a Zipf curve, as real chat, ticket and order data does: the busiest channel has 12.3% of all messages, and the top 10 have 35.8%. The core of the router is short:

def keyspace_id(key: int) -> bytes:
    # Stand-in for Vitess's hash/xxhash vindex: any stable, well-mixed 64-bit hash works.
    return hashlib.blake2b(str(key).encode(), digest_size=8).digest()

def port_for(self, key):
    ksid = keyspace_id(key)
    for name, port in shards.items():      # {"-40": 3311, "40-80": 3312, ...}
        if in_range(ksid, name):
            return port

All five servers share one machine and one SD card, so the absolute numbers are higher and noisier than on real hosts, and every shard competes for the same four CPU cores. Read the ratios, not the milliseconds.

1. The sharding key decides the price of every query

We measured two access patterns. A fetches one message by id, from the same 2,000 random ids in all three variants: with the channel known (targeted), through a lookup vindex, or by scattering to every shard. B fetches the latest 50 messages, by channel (the sharding key, with channels picked by the same Zipf weights as the data) or by user (not the sharding key, so scatter and merge). For latency, each type ran 1,000 times, interleaved in random order after a warm-up. For throughput, each type ran for 15 seconds on 8 client threads:

One message by id: queries per second on 4 shards

8 client threads for 15 s per query type, the same 2,000 message ids in all three cases, four MySQL 8.4 shards and the router on one Raspberry Pi 4. Total shard queries per second stayed between 1,194 and 1,419 in all three cases; the difference is how many of them each read needs.

The latency gap is modest because the router asks the shards in parallel. The throughput gap is not. Whatever the query type, the whole setup handled between 1,194 and 1,419 shard queries per second. A scatter read spends four of them, so the same setup serves about a quarter as many reads. Add shards and a scatter read gets more expensive, not less, which is the opposite of why you sharded.

Be clear about what the ceiling is here: the router, its 8 client threads, its 8-worker scatter pool and all four MySQL servers share one 4-core Pi, so we measured the capacity of that whole system, not of MySQL alone. On separate hosts the arithmetic is the same in kind: every scatter read still takes a query slot on every shard host.

Ordered scatter queries can also over-fetch. "Latest 50 by user" asks each shard for up to 50 rows, so for a busy user it can fetch 200 and discard 150 in the merge. (Our users average 40 messages each, so most of these reads returned fewer.) Vitess has to do the same for ORDER BY ... LIMIT across shards.

This is how a scatter query took Slack down. In its February 2022 incident, a client boot query read a keyspace "sharded by user" without the sharding key, and Vitess ran it as a scatter query, which "sends the query to every shard". When a cache problem pushed more of those reads to the database, every shard took the load at once.

2. A lookup vindex buys part of it back

Sometimes you need to find a row by something other than the sharding key: a message by its id, an order by its number. Vitess's answer is a lookup vindex, a table that maps the other key to the keyspace ID. In our router it is message_lookup, sharded by message id, holding the channel:

def by_id_lookup(self, mid):
    row = self.run(self.port_for(mid), "SELECT channel_id FROM message_lookup WHERE id=%s", (mid,))
    return self.run(self.port_for(row[0][0]), f"SELECT {COLS} FROM messages WHERE id=%s", (mid,))

Two shard queries instead of four took throughput from 307 to 710 reads per second: 2.3x the scatter version, but still only 59% of the targeted read, because the lookup is a second round trip. The cost moves to writes: the Vitess docs say a lookup vindex needs "an extra write to the lookup table for insert and delete operations". In our demo that write usually lands on a different shard than the message, so keeping the two consistent is its own problem; Vitess documents consistent lookup vindexes for that. Add lookup vindexes for the few access paths that are hot, not for every column.

3. Cross-shard transactions without two-phase commit lose data

Once data lives on several servers, a transaction that touches two shards is two transactions. Vitess's default transaction mode, multi, commits them one after the other with, in the docs' words, "no atomicity". We tested what that means with credit transfers between accounts. With a 2% probability per transfer, we simulate a failure between the two steps: the script commits the debit and rolls back the credit, as if the process had died after the first commit. For single-shard transfers, the same failure means rolling back the one transaction before it commits. This is a simulation of the failure, not a crash-recovery test.

{
  "same-shard":  { "transfers": 2000, "injected_failures": 32, "credits_before": 1000000,
                   "credits_after": 1000000, "credits_lost": 0, "p50_ms": 3.26, "p99_ms": 484.49 },
  "cross-shard": { "transfers": 2000, "injected_failures": 52, "credits_before": 1000000,
                   "credits_after": 999480, "credits_lost": 520, "p50_ms": 5.91, "p99_ms": 467.95 }
}

Every simulated cross-shard failure lost money: 52 failures (2.6% of transfers in this run), 520 credits gone, because the debit had committed and the credit never did. On one shard, the 32 failures rolled back cleanly. The cross-shard version also took longer at the median, 5.9 ms against 3.3 ms, because it waits for two commits.

Vitess has an atomic two-phase commit mode (TWOPC). When the Vitess 21 announcement reintroduced it in 2024 it was "currently in an experimental state"; check the current docs for its status. The docs also warn that it "does not provide isolation", so other readers can see a half-finished transfer, and that it adds commit latency. Teams that have written about sharding with Vitess designed around cross-shard transactions. Shopify's Shop app removed all cross-shard transactions as part of sharding by user id. Cash App kept each customer's "entity group" together on one shard through every split. The rule that falls out: pick a sharding key that keeps the rows a transaction touches on the same shard, and use an idempotent, retried workflow or two-phase commit, with its costs, for the rest.

4. Resharding online, without the night of downtime

This is the operation that cost YouTube a night. Vitess now does it online with VReplication: copy the rows in chunks from a consistent snapshot, stream the changes that happen meanwhile from the binlog, compare both sides, then switch traffic. We did an insert-only version in miniature, splitting c0- into c0-e0 (staying on port 3314) and e0- (moving to the spare on 3315). All three tables moved: messages (keyed by channel), the lookup table (keyed by message id) and accounts. A writer thread inserted messages and their lookup rows the whole time, one at a time with a 5 ms pause between them. A reader thread looked up random seeded messages through the lookup vindex and checked that each was found, and ran scatter reads by user and checked them for duplicate rows:

1. Mirror: inserts for the moving range e0- go to both the old and the new shard. This stands in for VReplication's change stream, and it is the same double-write approach Slack used for its migration. Unlike VReplication it handles only inserts: no ordering of updates and deletes, and no replay after a crash.
2. Copy: existing e0- rows move in chunks of 2,000. These are plain reads, not a consistent snapshot, which is safe here only because rows are never updated.
3. Verify: for messages, the row count and an XOR of per-row CRC32 values (over id, channel, user and body) up to a watermark id must match on both sides; for the lookup and accounts rows, the count and a hash of the sorted rows. This is the job VDiff does, more thoroughly.
4. Switch: one atomic change to the router's in-memory shard map routes e0- to the new server. Scatter reads keep only rows inside each shard's range, so rows still sitting on the old shard are not returned twice. The script then waits 3 seconds before cleanup instead of draining in-flight requests, so the safety of the hand-over depends on timing; Vitess fences and buffers traffic during SwitchTraffic instead.
5. Clean up: delete the moved rows from the old shard, then check the end state.

From results/reshard.json:

copy            44,937 messages, 24,796 lookup rows, 129 accounts in 23.4 s
verify          44,963 messages up to the watermark: count and CRC32 XOR match;
                lookup and accounts rows identical
map update      0.026 ms (the in-process shard-map swap, not a client cutover)
cleanup         moved rows deleted from the old shard in 23.7 s
writer          994 inserts during the run, 0 errors
reader          2,158 lookups all found; 2,158 scatter reads, no duplicates
end state       every expected message id exactly once (200,994), 200,994 lookup
                rows, 1,000 accounts, 0 rows on the wrong shard or with a wrong ksid

Nothing was lost and no write failed. Because the writer waited for each insert before sending the next, a slower write simply meant fewer writes; this does not show how a system with a fixed incoming rate would queue. The cost showed up as latency: the writer's median insert (a message plus its lookup row, across all shards) went from 8.3 ms before the move to 16.5 ms during the copy and 21.3 ms during the cleanup. The copy and the deletes most likely competed with it for the shared SD card. Vitess ships a throttler that VReplication uses to limit exactly this kind of impact.

For comparison, the simple way is to stop writes, copy, switch and start again. We did not measure that. Our online copy took 23.4 seconds for about 70,000 rows while sharing the machine with the writer and reader, and copy time grows with the size of the shard.

5. More shards do not fix a hot key

Hashing spreads keys evenly, not load. Every message in a channel has the same sharding key, so a busy channel is a busy shard, however many shards you have. We took the real per-channel row counts and worked out the busiest shard's share for 2 to 128 equal shards:

Share of all messages on the busiest shard

200,000 messages, 2,000 channels with Zipf activity; the busiest channel holds 12.3%. Equal keyspace ranges. 'Hot channel bucketed' splits that one channel's messages over 16 sharding keys.

At 128 shards, where an even split would put 0.78% on each, the busiest shard still held 14.6%, because the busiest channel alone is 12.3%. We also modeled splitting that one channel's messages evenly over 16 sharding keys (for example, channel plus message_id % 16). That brought the busiest shard down to 8.3%, and the next-busiest channel, 6.1% of messages, became the new floor. Bucketing has its own cost: a read of the whole channel now has to ask up to 16 places and merge, which is the scatter problem from section 1 on a smaller scale.

Slack wrote up a related failure in 2023. In The Query Strikes Again, about an incident in October 2022, one shard "contained 6% of the user's subscription data", and write load on it made the tablets run out of memory until "the kernel OOM-killed the MySQL process." That was a concentration of load on one shard during a bulk operation, not necessarily a single hot key, but the shape is the same: one shard takes far more than its share. The Vitess docs' advice for hot shards is to split shards, and that works for a hot range of many keys. For a single hot key, the fix is in the data model: bucket the key, or pick a different one.

When to shard, and how to pick the key

• Do the cheap things first. Read replicas for read load. Functional partitioning by domain, as GitHub did, buys a lot of headroom before you need horizontal sharding at all, and it is easier to undo.
• Shard when one primary cannot take the writes or hold the data. That is the point YouTube, Slack and GitHub each reached.
• Pick the key from your most common queries. PlanetScale's guidance is to optimize for "the most common queries with the goal of avoiding cross-shard and cross-keyspace queries". Our numbers show why: a query with the key used 1 shard query, one without it used all of them.
• Keep transactions on one shard. Choose a key that groups the rows a transaction touches, like Cash App's entity groups, and do the rest with idempotent workflows.
• Measure key skew before you choose. Count rows and traffic per key value. If one value is a large share, consider bucketing it, and weigh that against the extra fan-out on reads of that key.

Vitess, PlanetScale and TiDB

• Vitess is the open source layer all of this comes from, a CNCF graduated project since November 2019. You run VTGate, VTTablet and the MySQL servers yourself, often on Kubernetes. HubSpot reported over 750 Vitess shards per data center, each a 3-instance MySQL cluster, all inside Kubernetes.
• PlanetScale runs Vitess as a managed service and says it builds its product "on top of Vitess, which provides a transparent sharding solution for MySQL". You get the same primitives, keyspaces, vindexes and online resharding workflows, without operating the layer.
• TiDB takes a different route. It is a distributed SQL database with a MySQL-compatible protocol, and its storage layer, TiKV, splits data into Regions automatically: when a Region grows past 384 MiB by default, it splits. You do not choose a sharding key or run reshards yourself. We did not test how it handles a single hot key.

What we could not conclude

• Absolute performance. Five servers on one Raspberry Pi with an SD card is not a production fleet. Commits wait on the card, which put write p99 in the hundreds of milliseconds throughout, before, during and after the reshard.
• Vitess's own overhead. Our router is Python, not VTGate. Real VTGate adds query planning and connection pooling, and we did not measure how its scatter queries compare with ours.
• Large reshards. We moved about 45,000 messages, or 70,000 rows across three tables. At hundreds of millions, copy throughput, replica lag and throttling dominate, and we did not test them.
• Real two-phase commit. We tested the default mode's failure, not Vitess's atomic mode.

Summary

• Sharding is a routing problem first. A query the router can send to one shard costs one shard; a query it cannot narrow down costs all of them. On our four shards, that was 1,194 against 307 reads per second for the same message.
• A lookup vindex more than doubles the throughput of the scatter read for one access path, at the price of an extra write and a second round trip.
• Without two-phase commit, a cross-shard transaction that dies between commits leaves partial data every time. Design keys so transactions stay on one shard.
• Online resharding (mirror, copy, verify, switch) moved about 70,000 rows across three tables with zero write errors and zero lost rows, while lookups kept finding their rows. The cost was higher insert latency while it ran.
• Hashing does not fix a hot key. Measure skew before you pick the key, and bucket hot values.

Most of that move happened earlier than people think. In 26.04 LTS most tools already come from uutils, so the question for most teams is narrower: what is different in practice, and what will your scripts notice? We ran 162 shell test cases in chroots of the Ubuntu 24.04, 25.10, 26.04 LTS and 26.10 beta base images, and in 26.10 we ran each one twice: once with uutils and once with the GNU binaries Ubuntu still ships. Same files, same OS, only the implementation changed.

TLDR

• On the same 26.10 system, 150 of 162 test cases gave byte-identical captured stdout, stderr and exit codes with GNU and uutils.
• The other 12 come down to four changes that can alter what a script does, plus five error messages:
  • ls --version now says "uutils coreutils", so scripts that look for "GNU" to pick GNU or BSD flags take the wrong branch.
  • cut -c counts characters, not bytes, on UTF-8 text. This is new in 26.10; 26.04 still counts bytes.
  • timeout -s KILL exits with 124 instead of 137.
  • base64 -d printed nothing for our invalid input, where GNU printed the bytes it decoded before failing.
• The rest are message text: different wording and quote marks in errors. They matter only if a script greps stderr.
• If you are on 26.04 LTS, most of this already applies to you. 26.10 changes 6 of our cases: the version strings, cut -c, and three error messages.
• 25.10 shipped real bugs that are gone now: comm dropped a line in up to 19 of 20 runs, and date -r ignored the file.
• Speed, on our Raspberry Pi 4: plain sorts and counting got faster (sort about 16 to 19%), hashing is the same, and multi-key sorts and walking many small files got slower (sort -k 35%, copying a 5,000-file tree 61%, ls -lR twice as slow).
• Two other 26.10 changes servers will notice: dbus-broker replaces dbus-daemon, and the default OpenSSH client no longer understands GSSAPI options.

Prerequisites

• A Linux machine with sudo, curl, Python 3 and about 2 GB of free disk if you want to run the tests yourself. No Docker is needed: the scripts use chroot on Ubuntu's base images. They download the arm64 images (we ran them on a Raspberry Pi 4); change the file names in scripts/01-fetch.sh for amd64.
• The repo:

The-DevOps-Daily/ubuntu-2610-coreutils-check on GitHub

What changed, release by release

We checked each base image with dpkg-query and by following the /usr/bin links (results/inventory.txt in the repo):

The GNU code has not gone away. Every release since 25.10 ships the gnu-coreutils package, with each tool installed under a gnu prefix: gnuls, gnucp, gnusort. In 26.10 that is 104 binaries, and it is what made a clean comparison possible.

ubuntu 26.10 beta

# ls --version | head -1
ls (uutils coreutils) 0.12.0
# gnuls --version | head -1
ls (GNU coreutils) 9.10
# readlink -f /usr/bin/cp
/usr/lib/cargo/bin/coreutils/cp

How we tested

Each test is one line in cases.tsv, run with bash -c in a fresh copy of the same input files, with LC_ALL=C.UTF-8 and TZ=UTC. The "releases" are chroots of Ubuntu's arm64 base images on one Raspberry Pi 4 host (kernel 6.12), with the host's /dev mounted read-only and a fresh /proc, running as root. That is enough for coreutils, which is userspace, but it says nothing about kernel, service or permission behavior. We wrote the cases from what shell scripts actually do: ls -l with time styles, sort -h, -V and -k, date -d parsing, stat -c, cut, tr, sha256sum -c, timeout, mktemp, install -D, dd, error messages for missing files and bad options, and so on.

We ran all 162 cases in five setups:

• gnu-24.04: Ubuntu 24.04.5, GNU 9.4.
• mixed-25.10: Ubuntu 25.10.
• mixed-26.04: Ubuntu 26.04.1 LTS.
• uutils-26.10: Ubuntu 26.10 beta, as shipped.
• gnu-26.10: the same 26.10 rootfs with a directory of links to the gnu* binaries first in PATH.

A case "differs" when its captured stdout, stderr or exit code is not byte-identical. Some cases filter output through awk, head or sed, or print a tool's exit code with echo "rc=$?" (so the captured exit code is the shell's), and file side effects are only checked where a case looks at them. Bash builtins (echo, printf, test) would hide the coreutils binary, so those cases call the binary through env.

gnu-26.10  vs uutils-26.10   12 of 162 differ   (same OS, only the implementation)
gnu-24.04  vs uutils-26.10   13 of 162 differ   (an LTS user jumping to 26.10)
gnu-24.04  vs mixed-26.04    12 of 162 differ   (LTS to LTS)
mixed-26.04 vs uutils-26.10   6 of 162 differ   (what 26.10 adds)
gnu-24.04  vs gnu-26.10       4 of 162 differ   (GNU 9.4 to 9.10 on its own; one is only a PID in a bash message)

The four changes that can change what a script does

1. Scripts that detect GNU now take the BSD branch

Portable scripts often check whether they run with GNU or BSD tools, because the flags differ (date -d vs date -v, stat -c vs stat -f). The most common check reads the version string:

if date --version 2>/dev/null | grep -q GNU; then
  tomorrow=$(date -d tomorrow +%F)      # GNU flags
else
  tomorrow=$(date -v+1d +%F)            # BSD and macOS flags
fi

On 26.10, date --version prints date (uutils coreutils) 0.12.0 (on 26.04, 0.8.0). The grep fails, the script takes the BSD branch, and date -v+1d fails with an unknown-option error. Our detect-gnu-date and detect-gnu-stat cases printed gnu-path on 24.04 and bsd-path on 25.10, 26.04 and 26.10.

uutils implements the GNU flags, so the fix is to test for the feature, not the vendor:

if date -d tomorrow +%F >/dev/null 2>&1; then
  tomorrow=$(date -d tomorrow +%F)      # GNU-style date: GNU or uutils
else
  tomorrow=$(date -v+1d +%F)            # BSD and macOS
fi

Search your repos for --version | grep and --version 2>&1 | grep -q GNU. This one is easy to miss, because nothing errors where the check runs. The script fails later, in a branch that was written for a Mac.

2. cut -c counts characters now (new in 26.10)

cut -c1-4 on UTF-8 text

# input: 'héllo wörld' and 'naïve café'
# GNU 9.4, GNU 9.10, and uutils 0.8.0 on 26.04
$ cut -c1-4 utf8.txt
hél
naï
# uutils 0.12.0 on 26.10
$ cut -c1-4 utf8.txt
héll
naïv

GNU cut treats -c as bytes, so é (two bytes in UTF-8) takes two of the four positions. POSIX says -c selects characters, and uutils 0.12.0 now does that, so the new behavior is the standard one. Either way, the output changed under the same command, and 26.04 LTS still behaves like GNU. If a script slices fixed-width fields out of text that can contain non-ASCII names, paths or log messages, the columns shift. Use cut -b when you mean bytes.

3. timeout -s KILL exits with 124, not 137

$ timeout -s KILL 1 sleep 5; echo "rc=$?"

GNU 9.4 / 9.10:   bash: line 1: ... Killed   timeout -s KILL 1 sleep 5
                  rc=137
uutils (25.10+):  rc=124

GNU documents this in timeout --help: it exits with 137 "if COMMAND (or timeout itself) is sent the KILL (9) signal (128+9)". uutils returns 124. A wrapper that treats 124 as "timed out" and anything else as "the command failed" now reports a timeout where it used to report a failure, or the other way around. Check for both. The default signal (TERM) returned 124 in every setup, and --preserve-status behaved the same everywhere.

4. base64 -d on bad input prints nothing

$ echo 'not*base64' | base64 -d; echo "rc=$?"

GNU:     <two garbage bytes>rc=1      stderr: base64: invalid input
uutils:  rc=1                         stderr: base64: error: invalid input

Both fail with exit code 1. GNU writes the bytes it managed to decode before it hit the bad character, and for this input uutils wrote nothing. We tested one short malformed input, so do not assume either tool decodes all-or-nothing on long streams. This only matters if something downstream reads partial output, but that is exactly the kind of thing that turns up in a script that pipes base64 -d into a file without checking the exit code.

Error messages: different text, same exit codes

The other differences are wording. Exit codes matched in every case below, so they only affect scripts that parse stderr or tests that compare it:

The quote marks are worth knowing about: GNU uses Unicode curly quotes in a UTF-8 locale, so a grep "invalid date 'not" in a test suite matched nothing on GNU and matches now.

A change that is not about Rust: mv -n

One difference shows up between GNU versions, before uutils is involved. On 24.04 (GNU 9.4), mv -n onto an existing file prints not replacing and exits 1. GNU 9.5 and later (see the 9.5 release announcement), and uutils, skip silently and exit 0. If you move from 24.04 to 26.04 or 26.10 and a script used mv -n ... || handle_conflict, the handler no longer runs.

25.10 shipped bugs that 26.04 fixed

The 25.10 results are a reminder of why interim releases exist. With uutils 0.2.2, 25.10 differed from 24.04 in 17 cases, and some were plain bugs:

• comm lost a line. comm <(sort dup.txt | uniq) <(printf 'apple\ndate\n') dropped cherry from the output. It is timing-dependent: in our batches of repeats it dropped the line 7 and 19 times out of 20 on 25.10, and never on 26.04 or 26.10 (results/comm-repeat.txt has the last batch).
• date -r FILE ignored the file and printed the current time instead of the file's modification time.
• Error messages for cat and sha256sum on a missing file ended with Rust's (os error 2).

All three are fixed in 26.04 and 26.10, except that stat on a missing file still printed (os error 2) in 26.04; 26.10 fixed that too. 25.10 also gave a different order from GNU for shuf --random-source with the same source bytes. That is a compatibility difference rather than a bug, since nothing promises two implementations map random bytes the same way, and 26.04 matches GNU again. If you skipped 25.10, you never saw most of this.

Speed

We timed 15 workloads with GNU 9.10 and uutils 0.12.0 in the same 26.10 rootfs on a Raspberry Pi 4 (4 cores, 4 GB): 9 runs each, alternating the two implementations, inputs already in the page cache, the first run dropped as a warm-up. The inputs were a million random integers (10 MB), a million lines of text (19 MB), a 100 MiB binary file and a tree of 5,000 small files. Each pair ran GNU first, then uutils. The harness records time only: it does not check exit codes or compare outputs, so treat these as timings of the same commands, not a validated benchmark suite.

uutils 0.12.0 run time as a share of GNU 9.10 (lower is faster)

Median of 8 timed runs per workload, Ubuntu 26.10 beta rootfs, Raspberry Pi 4. 100% = same speed as GNU. Raw runs in results/bench.tsv.

On this machine and these inputs the spread between runs was small (results/bench-summary.tsv has the ranges), and the pattern is consistent: plain sorts and counting are faster with uutils, hashing is the same, and multi-key sorts and anything that walks many small files are slower. Copying and deleting the tree took 61% longer, ls -lR twice as long. Results on x86 servers and other disks can differ. For most scripts the absolute numbers are small. A backup or deploy script that copies a large tree of small files is where you would notice.

A note on method: our first benchmark run wrote every command's output to a regular file, because the chroot had no /dev and >/dev/null created one. That made output-heavy commands look slow on both sides. We threw those numbers away and reran with /dev mounted; the scripts in the repo do this.

Two more 26.10 changes that servers notice

dbus-broker replaces dbus-daemon

Canonical announced in June that 26.10 replaces dbus-daemon, the default since 2004, with dbus-broker for both the system bus and user session buses, on fresh installs and upgrades. The announcement calls it "a drop-in replacement. Same protocol, same config files, same bus policies." The archive shows the switch:

                          26.04 LTS                 26.10 beta
provides the system bus   dbus                      dbus-broker
dbus-broker               universe                  main
dbus, dbus-daemon         main                      universe

For most servers nothing visible changes. The thing to check is a container image or CI job that installs dbus by name to get a session bus with dbus-run-session (headless browser tests often do). In 26.10, apt-get install dbus still pulls in dbus-daemon, but both are in universe now, so an image with universe disabled cannot install them. The announcement also documents how to switch back.

The default OpenSSH client no longer understands GSSAPI

The release notes say OpenSSH has been split into openssh and openssh-gssapi, and that fresh installs get the version without GSSAPI and Kerberos. We installed the default client in each release and gave it the GSSAPI options that many existing configs carry:

26.04 (OpenSSH 10.2p1):  no GSSAPI warning, exit 0
26.10 (OpenSSH 10.5p1):  /root/.ssh/config line 2: Unsupported option "gssapiauthentication"
                         /root/.ssh/config line 3: Unsupported option "gssapidelegatecredentials"
                         exit 0

We tested this with ssh -G, which evaluates the configuration without connecting. It warns and carries on, so a config with these lines still loads, and every ssh run that reads them will print the warning on stderr. Kerberos authentication is gone with the GSSAPI build, as the release notes say; we did not test a real Kerberos login. The default /etc/ssh/ssh_config in 26.04 had GSSAPIAuthentication yes enabled; in 26.10 that line is gone. If you use Kerberos for SSH, install openssh-client-gssapi (and openssh-server-gssapi on servers). If you do not, remove the GSSAPI lines from configs you copy onto new machines, or the warnings will end up in every log of every Ansible run.

How to check your own scripts before you upgrade

1. Run them against 26.10 without upgrading anything. The repo downloads Ubuntu's base image and runs commands in a chroot. Put your script in rootfs/26.10/work and run it with sudo chroot rootfs/26.10 /bin/bash -c 'cd /work && ./your-script.sh'. Mount /dev and /proc first, as scripts/03-run-cases.sh does.
2. Compare with GNU on the same box. Put the gnu* binaries first in PATH with a directory of links (see scripts/02-extract.sh), run the script again, and diff the output.
3. Grep for the patterns above: --version | grep, cut -c on text that may hold non-ASCII, timeout -s KILL, and error-message matching in tests.
4. Switch back if you need to. Ubuntu's 26.04 notes give the commands. They also apply to 26.10:

# to GNU coreutils
sudo apt install coreutils-from-gnu coreutils-from-uutils- --allow-remove-essential
# back to Rust coreutils
sudo apt install coreutils-from-uutils coreutils-from-gnu- --allow-remove-essential

For a single script, calling gnudate, gnucut or gnutimeout directly is enough.

What we could not conclude

• Locales other than C.UTF-8. The base images ship no other locales, and sort and ls ordering depend on the locale. Test en_US.UTF-8 on a full install if your scripts depend on sort order.
• Long-running and terminal behavior. We did not test tail -f, stty, dd progress output or anything that needs a TTY.
• Your edge cases. 150 of 162 identical test cases is a good sign, not a guarantee. The cases are in a text file in the repo; add yours and send them back.

Summary

• 26.10 finishes a move that 26.04 LTS mostly made: cp, mv and rm are Rust now, and uutils jumps from 0.8.0 to 0.12.0.
• On the same system, 150 of 162 shell test cases gave identical captured results with GNU and uutils.
• Fix these before you upgrade: GNU detection by version string, cut -c on non-ASCII text, scripts that expect 137 from timeout -s KILL, and anything that parses error messages.
• Moving from 24.04, also check mv -n: its exit code changed in GNU itself.
• Check container images that install dbus by name (it is in universe now), and remove GSSAPI lines from SSH configs unless you install openssh-client-gssapi.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Introducing DigitalOcean Insights: Built-in Observability for Your Agents, Applications, and Infrastructure

Coding agents have changed how fast teams ship, and more of the software they ship now includes agents. A production agent spans model calls, routing decisions, tools, and the infrastructure underneat

📅 Oct 12, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 Who owns NIS2 and DORA on a Kubernetes platform team

There is a meeting that repeats across a lot of European engineering organizations right now. Someone from legal or risk brings a NIS2 control, a DORA article, or a “can we evidence this” request into

📅 Oct 9, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Actionable disconnected cluster analysis, available now with pen drive powered by Red Hat Lightspeed

Red Hat OpenShift doesn't stop at the cloud boundary. Telecommunications providers, government agencies, and financial institutions run OpenShift in environments that are intentionally air-gapped — an

📅 Oct 8, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 The Shift to cgroup v2 in Kubernetes: What You Need to Know

In Linux, cgroups (control groups) are a kernel feature used for managing system resources. Kubernetes uses cgroups to allocate resources like CPU and memory to containers, ensuring that applications

📅 Oct 6, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Scaling Kubernetes Workloads with Node Swap

Memory is often the first hard limit a Kubernetes cluster hits. Nodes run out of RAM long before they run out of CPU, and the new wave of agentic AI workloads makes this worse. These workloads demand

📅 Oct 5, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Docker Hub security advisory: personal access tokens transferred during account-to-organization conversions

CVE-2026-105445: Docker Hub account conversion could re-assign personal access tokens to a new owner. Read what happened, who was affected, and the fix.

📅 Oct 12, 2026 • 📰 Docker Blog

🔗 Read more

📄 From zero to agent builder: Running kagent on your laptop

If you have seen a demo of a no-code agent builder, you know the moment. Someone types one sentence, waits a minute, and there it is: a new agent, running, with tools. It is impressive. For...

📅 Oct 12, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Join Platform Engineering Day at KubeCon + CloudNativeCon North America 2026

As organizations accelerate their adoption of AI and cloud native technologies, platform teams are being asked to solve a growing set of challenges. How can developers access new capabilities such as

📅 Oct 9, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Making the Golden Kubestronaut a little more golden

When I became a Golden Kubestronaut, I wanted to celebrate the achievement in a way that felt lasting. Like most technical achievements, certifications usually live digitally. You receive the badge, u

📅 Oct 9, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Harness CI Pipeline Tutorial: Build and Push to Docker

Step-by-step: connect GitHub, let Harness AI generate a CI pipeline for your Java project, add Docker push. Full pipeline ran in 3 minutes. | Blog

📅 Oct 8, 2026 • 📰 Harness Blog

🔗 Read more

📄 Docker Registry: Public, Private, and Pull-Through caching

Harness Artifacts provides a secure, private, and cloud-native Docker registry where developers can store, proxy, and manage Docker images with enterprise-grade | Blog

📅 Oct 8, 2026 • 📰 Harness Blog

🔗 Read more

📄 OpenTelemetry Community Awards 2026

We are delighted to announce the 3rd annual OpenTelemetry Community Awards this year! This project and its newly minted status as a graduated CNCF project would not be possible without many, many awes

📅 Oct 7, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

🔄 CI/CD

📄 GitLab Critical Patch Release: 19.4.2, 19.3.4, 19.2.8

📅 Oct 12, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Hack the World: Why hackathons are still the best place to learn to build

The barriers to building software have collapsed. Today, anyone can build. Hackathons are a great place to start learning. The post Hack the World: Why hackathons are still the best place to learn to

📅 Oct 9, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Gitea Runner 5.0.0 is released

We are happy to announce the release of Gitea Runner 5.0.0. This is a quiet major release. A handful of small breaking changes required the version bump, but most setups upgrade without changes. T

📅 Oct 8, 2026 • 📰 Gitea Blog

🔗 Read more

📄 How one bug bounty researcher chooses the features they investigate

As we kick off Cybersecurity Awareness Month, the GitHub Bug Bounty team spotlights @vaib25vicky, exploring their methodology, techniques, and experiences hacking on GitHub. The post How one bug bount

📅 Oct 8, 2026 • 📰 GitHub Blog

🔗 Read more

📄 How to Deploy Your First App with Harness CD in 15 Minutes

A first deployment walkthrough for Harness CD using a real sample app. | Blog

📅 Oct 8, 2026 • 📰 Harness Blog

🔗 Read more

📄 Track organization-wide security risk in one dashboard

If you run application security across more than one top-level group, getting a single organization-wide view of your risk has meant manually pulling together data. That is operational work rebuilt in

📅 Oct 8, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Secret protection must scale with software

Developers aren’t becoming more careless; they’re being outpaced. The tools that let developers create more software should also take on more of the work of protecting it. The post Secret protection m

📅 Oct 7, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Building Git infrastructure for agent-scale development

We're rebuilding GitHub's Git infrastructure while GitHub keeps running, creating a foundation for agent-scale software development. The post Building Git infrastructure for agent-scale development ap

📅 Oct 6, 2026 • 📰 GitHub Blog

🔗 Read more

📄 GitLab Transcend: Speed you can trust, all the way to production

At Transcend, we made over a dozen announcements across all four layers of our architecture for agentic software engineering: agent orchestration, data and context, DevOps workflows, and governance an

📅 Oct 6, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Dependency Firewall: Block risky packages before the build

Attackers disguise malicious packages as ones you trust. In June 2026, GitLab researchers found five malicious PyPI packages, four of them typosquats of Flask, Requests, and NumPy, that run at install

📅 Oct 6, 2026 • 📰 GitLab Blog

🔗 Read more

🏗️ IaC

📄 Amazon Bedrock now supports reasoning summaries for OpenAI models

Amazon Bedrock now supports the reasoning.summary parameter for OpenAI models through the Responses API. This capability lets you request a human-readable summary of a model’s reasoning alongside its

📅 Oct 9, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Concurrent change requests in Pulumi ESC

With ESC Approvals, you can propose, review, and apply environment changes with change requests. Until now, an environment could only have one open change request at a time. If a teammate’s change was

📅 Oct 7, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Pulumi Google Cloud Provider Version 10.0.0

TL;DR: we are releasing a new major version of the Google Cloud provider. To migrate, ask Neo or your coding agent to run the Pulumi provider-upgrade skill (how to use it) or follow the v10 migration

📅 Oct 6, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 Introducing DigitalOcean Signals and Simulations: see how your agents and models really behave in production, then test the fix before it ships

You ship an agent. It mostly works. Then the feedback loop becomes a support ticket: a user got the wrong answer, a tool call looped four times, a refund went out that shouldn’t have. Standard observa

📅 Oct 12, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 Prometheus and OpenTelemetry interoperability in 2026: Survey results

We ran a survey asking users of OpenTelemetry and Prometheus how they collect, process, and store metrics. The goal was to understand, with real usage data rather than assumptions, how far the ecosyst

📅 Oct 12, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 OTel-Native by Design - Building Products That Export to Any Observability Stack

With contributions from Dan Gomez Blanco (New Relic). If you’re building self-hosted software or a SaaS product, your users will eventually ask to send logs, traces, and metrics to their own observabi

📅 Oct 9, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 How New Relic engineers reliability at scale

Learn how New Relicengineers reliability at scale with telemetry, resilient architecture, and automation to detect, contain and respond to failure.

📅 Oct 9, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Sentry JavaScript SDK v11 adds tracing for all JS platforms

Sentry's v11 JavaScript SDKs trace dependencies with a code transformer instead of OpenTelemetry instrumentation, adding Cloudflare Workers, Bun, and Deno

📅 Oct 9, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Take Grafana Labs' 5th annual Observability Survey

Over the past year, "agentic" has gone from an emerging buzzword to an integral part of worklife for many engineers—and that's just scratching the surface on all that's changed. With so much upheaval,

📅 Oct 8, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Manage your OpenTelemetry Collectors with Fleet Management in Grafana Cloud

If you’ve built your telemetry pipelines around OpenTelemetry Collectors, you’ve already invested in a collector distribution, YAML configuration, and a deployment model that fits your infrastructure.

📅 Oct 8, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Reducing Android scope-sync overhead in Sentry Flutter

How Sentry Flutter uses a worker isolate for Android SDK operations and reduces duplicate normalization and JSON encoding overhead.

📅 Oct 8, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Manage synthetic checks at scale: Introducing folders in Grafana Cloud Synthetic Monitoring

As your use of Grafana Cloud Synthetic Monitoring grows, so does the number of checks you need to manage across services, environments, and teams. Eventually, a single flat list of checks becomes diff

📅 Oct 7, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Introducing AI Evaluation: Close the Loop Across the Full AI Lifecycle

Discover New Relic AI Evaluation: Bridge pre-production prompt testing with live transaction telemetry to evaluate quality, reduce costs, and scale GenAI.

📅 Oct 6, 2026 • 📰 New Relic Blog

🔗 Read more

📄 The Velocity Dilemma in Modern Software Delivery

LaunchDarkly and New Relic close the loop between release and observability, automating rollbacks before incidents reach your users.

📅 Oct 6, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Introducing AI Observability for OpenTelemetry

Unify OpenTelemetry AI with New Relic. Track token costs, discover shadow AI, and block prompt injections in real time without data duplication.

📅 Oct 6, 2026 • 📰 New Relic Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Oct 12, 2026 • 📰 Linode Blog

🔗 Read more

📄 Agentic data science with the Data Agent Kit

Imagine your compliance team notices that payments are taking longer than expected to clear. They suspect fraud, and you’re the data scientist helping figure out which transactions to investigate firs

📅 Oct 12, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Developer-first innovation: Reducing technical debt with Red Hat Enterprise Linux

Managing software teams means balancing 2 conflicting priorities: developers want freedom to pick their tools, while leadership needs security and stability. When every team runs a different operating

📅 Oct 12, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 AWS Security Hub now exports findings to S3 in CSV or JSON format

Today, AWS Security Hub announces support for exporting findings to Amazon S3 in CSV or JSON (OCSF) format. Security teams that need findings outside the console for use cases such as compliance repor

📅 Oct 9, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 2026-10-07: PostgreSQL JDBC 42.7.14 Security update for multiple CVE's

The PostgreSQL JDBC team has released a security release for 2 CVE's CVE-2026-107314 and the GitHub Security Advisory GHSA-rhp9-mr79-r74h and CVE-2026-107315 and the GitHub Security Advisory GHSA-f64h

📅 Oct 9, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Why "secure by design" is the new standard for open source

Open source software faces significant regulatory shifts, making "secure by design" development practices increasingly important. The Cyber Resilience Act (CRA) takes effect on December 11, 2027, and

📅 Oct 9, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Harness acquires Augment Code to advance the Autonomous SDLC

Harness Cosmos Software Factory Agent automates engineering from idea to code, connecting code context with delivery, security, testing, and production workflow | Blog

📅 Oct 8, 2026 • 📰 Harness Blog

🔗 Read more

📄 Frontier models found the vulnerabilities. Only the attacker found the chains.

Static analysis found the flaws, but only live attack testing proved how they could be chained into breaches. A comparison of Evo COS, Claude Security, and Claude Code Security.

📅 Oct 7, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Red Hat OpenShift Platform Plus bundle available on hyperscaler marketplaces

Note: This post was updated on October 7th with the latest information.Organizations using Red Hat OpenShift cloud services are increasingly seeking ways to extend their platform’s capabilities with e

📅 Oct 7, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Autonomous Attacks Are Already Here. The Defense Has to Match Their Speed.

Autonomous attackers are shrinking the window for defense. Learn how continuous discovery, remediation, validation, and prevention can help security teams keep pace.

📅 Oct 7, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Gitea 28.1.0 is released

We are excited to announce the release of Gitea 28.1.0, the first maintenance release for the 28 series. It contains security fixes and fixes for regressions reported after 28.0.0, including pushe

📅 Oct 6, 2026 • 📰 Gitea Blog

🔗 Read more

📄 What’s new in security for Ubuntu 26.10?

Ubuntu 26.10 “Stonking Stingray” arrives at a time when AI tools are accelerating vulnerability discovery. Patching faster is essential, but the resilience of the underlying operating system must also

📅 Oct 6, 2026 • 📰 Ubuntu Blog

🔗 Read more

💾 Databases

📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers

Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.

📅 Oct 12, 2026 • 📰 Linode Blog

🔗 Read more

📄 Introducing TimesFM 3.0 with multivariate time-series forecasting in BigQuery

Overview Since introducing zero-shot time-series forecasting in BigQuery with AI.FORECAST — initially powered by TimesFM 2.0 and later upgraded to TimesFM 2.5 — you have been able to generate forecast

📅 Oct 12, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Filter, reshape, and redact logs before export with Transformers

Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export. Powered by Apache DataFusion and Apache Arrow, Transformers apply those changes di

📅 Oct 12, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Inside the Database Internals Talks at P99 CONF 2026

Since database performance is so near and dear to ScyllaDB, we reached out to our friends across the community to ensure that a nice range of distributed data systems, approaches, and challenges would

📅 Oct 12, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 Chaos engineering in practice: Lessons from enterprise customers

Working with enterprise customers on chaos engineering has reinforced 1 lesson: Resilience is a business requirement. Failed transactions interrupt revenue, infrastructure outages affect multiple work

📅 Oct 12, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 How we built the Redis Docs MCP for agents

Ask a coding agent how to configure eviction in Redis 8 and you'll usually get an answer. Less often than we'd like, that answer comes from authoritative docs. The agent either recalls something from

📅 Oct 9, 2026 • 📰 Redis Blog

🔗 Read more

📄 Redis OSS Is Now Available on AWS Marketplace

Redis Open Source Software (OSS) is now available through an official AWS Marketplace listing owned by Redis. The listing includes a Redis-published Amazon Machine Image (AMI), allowing teams to launc

📅 Oct 8, 2026 • 📰 Redis Blog

🔗 Read more

📄 pgx-bm25 1.0: BM25 ranked full-text search as a native PostgreSQL index

PGX Inc. has released version 1.0 of pgx-bm25, an extension that adds Okapi BM25 ranked full-text search to PostgreSQL as a native index access method. It supports PostgreSQL 17 and 18, and installs a

📅 Oct 7, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 pg_plan_filter 1.0.0 released

pg_plan_filter version 1.0.0 has been released. This loadable module test sstatements against specific configured criteria before execution, raising an error if the criteria are violated. This allows

📅 Oct 7, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 The OpenTelemetry Java agent 3.0 is almost here — preview it today

The 2.32.0 release of the OpenTelemetry Java agent is now out and serves as the release candidate for 3.0, which is targeted for October 2026. Some behavior may still change before 3.0, but you can pr

📅 Oct 6, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Why Open Source Still Matters When Agents Write the Code

Maintainers of well-known open source projects are dealing with a new kind of inbox. Pull requests still arrive, often in higher volume than ever, and many are clean, well-formatted, and plausible on

📅 Oct 6, 2026 • 📰 TiDB Blog

🔗 Read more

🌐 Platforms

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Oct 12, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Oct 12, 2026 • 📰 Linode Blog

🔗 Read more

📄 Trusted Data Apps for the Agentic Data Cloud, powered by Looker

As a business intelligence user, this probably sounds familiar: Most dashboards only provide you with a standardized canvas and a fixed library of charts, and the “analysis” consists of a CSV export t

📅 Oct 12, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 10 Years of the SRE book: Announcing the second edition of "Site Reliability Engineering”

A decade ago, Google released Site Reliability Engineering: How Google Runs Production Systems with O'Reilly Media. What began as a collection of our practices for managing large-scale infrastructure

📅 Oct 12, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Introducing DigitalOcean Agent Data Services: The Data Foundation for Agents

Agents are only as capable as the context they can reach. An agent with reliable access to the right information makes decisions grounded in fact; one without it guesses. And the work agents produce i

📅 Oct 12, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 DigitalOcean MicroVMs: Fast, isolated compute for your AI agent infrastructure

Coding-agent platforms, sandbox products, and code-execution services all need the same thing: isolated machines that start up fast, retain their state between bursts of work, and don’t consume comput

📅 Oct 9, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 Amazon EC2 R8gd instances are now available in additional regions

Amazon Elastic Compute Cloud (Amazon EC2) R8gd instances are available in AWS European Sovereign Cloud (Germany) region. These instances feature up to 11.4 TB of local NVMe-based SSD block-level stora

📅 Oct 9, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon EC2 R8g instances now available in additional regions

Starting today, Amazon Elastic Compute Cloud (Amazon EC2) R8g instances are available in the AWS European Sovereign Cloud (Germany) region. These instances are powered by AWS Graviton4 processors and

📅 Oct 9, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash

We are expanding the Clef decision model family with Clef-omni, natively processing audio, video, images, and text in a single pipeline. We’ve also lowered Clef-flash pricing and boosted Clef inferenc

📅 Oct 9, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects

On-demand CPU and memory profiling for Cloudflare Workers and Durable Objects is now available. Now you can generate interactive flamegraphs directly in production to quickly track down memory leaks a

📅 Oct 9, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Deno is joining Cloudflare

The Deno team is joining Cloudflare to radically simplify self-hosting Workers and Durable Objects, so developers can use the same primitives in more places.

📅 Oct 9, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 FinOps Was Built for Cloud. Here’s How It Adapts for AI

AI adoption is moving fast, and the teams responsible for managing its costs are trying to keep up. Between new models, agentic applications and changing pricing structures, even figuring out what the

📅 Oct 9, 2026 • 📰 Kubecost Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.142 (Insiders)

Learn what's new in Visual Studio Code 1.142 (Insiders) Read the full article

📅 Oct 14, 2026 • 📰 VS Code Blog

🔗 Read more

📄 Andrej Karpathy says only 5,000 people have seen what AI can really do

OpenAI founding member Andrej Karpathy, who joined Anthropic in May, thinks most of the world has little idea of what The post Andrej Karpathy says only 5,000 people have seen what AI can really do ap

📅 Oct 12, 2026 • 📰 The New Stack

🔗 Read more

📄 Build Your Game Challenge: Join in and Compete for Prizes

This October, we’re launching a new challenge for the JetBrains Academy community. To take part, complete our Build a Game With an AI Coding Agent course, customize the game with your own ideas, and s

📅 Oct 12, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Claude Code found my app’s shared contract in two MCP calls. Then the records ran out.

Every Claude Code user has watched a new session start from zero. You open it on a system you built The post Claude Code found my app’s shared contract in two MCP calls. Then the records ran out. appe

📅 Oct 11, 2026 • 📰 The New Stack

🔗 Read more

📄 Claude found 29,000 possible bugs in open source. Only 516 have been fixed.

Anthropic’s new OSS Scanner, launched last week as part of its broader Cyber Mission, exposes a problem inside the company’s The post Claude found 29,000 possible bugs in open source. Only 516 have be

📅 Oct 11, 2026 • 📰 The New Stack

🔗 Read more

📄 How to turn AI production feedback into better agents

Your agent is live. The service is healthy. But are its answers getting better? By connecting production traces, curated data, The post How to turn AI production feedback into better agents appeared f

📅 Oct 10, 2026 • 📰 The New Stack

🔗 Read more

📄 UI freeze reports are now available in JetBrains Marketplace

A user reports that the IDE stopped responding, and your plugin is the prime suspect. You try the same steps, but everything works. Without more details, such as a thread dump captured during the free

📅 Oct 9, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Mellum2.1 Gets to Work: A Fast Open Model for Coding Agents

Trained with reinforcement learning in real environments, Mellum2.1 is built for coding agents and fast sub-agents that run on your own hardware. Today, we’re releasing Mellum2.1, the next version of

📅 Oct 8, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Code Quality Never Changes – Or Is AI Challenging This Notion?

This post was adapted from Kai Schmithuesen’s talk at JetBrains Game Development Day 2026. See the full video presentation and Q&A below, or keep reading! First we are going to define what we mean by

📅 Oct 8, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Visual Studio Code 1.141

Learn what's new in Visual Studio Code 1.141 Read the full article

📅 Oct 7, 2026 • 📰 VS Code Blog

🔗 Read more

📄 Devpack for Rust: Zero to Rust in seconds

One of the best things about Rust is how easy it is to get started. It takes just a second to snap install rustup; rustup default stable, and from there you have a whole toolchain ready to compile Rus

📅 Oct 5, 2026 • 📰 Ubuntu Blog

🔗 Read more

Agent Droplets launched on 1 October 2026 as a public preview. The name suggests a server you rent. It is a monthly plan that prepays and discounts the usage you already pay for in Managed Agents. This post shows the plan math, the real cost of our runs, the break-even for two kinds of agent work, and a calculator you can run with your own numbers.

TLDR

• Pro costs $50 a month and covers $58.82 of usage at list price (15% off). Team costs $200 and covers $250 (20% off). After that, usage continues at list price.
• Pro beats pay as you go above $50 a month of discounted usage and saves at most $8.82. Team beats Pro above $208.82 and saves at most $50.
• The discount covers compute, memory, storage, tools, egress and DigitalOcean-hosted models such as GLM-5.3 and Kimi K3. Claude and GPT are not discounted.
• Our five recorded issue-to-PR runs were billed $0.064 per run, $0.107 per merged pull request. At that rate, Pro is cheaper from about 782 runs a month.
• One 39-minute interactive session on GLM-5.3 was billed $0.78, almost all of it tokens. At that rate, Pro is cheaper from 65 sessions a month.
• On our invoice, every token line of every session matched max(1 cent, amount rounded to the cent). That added 17% to our five runs and 25% to all 13 sessions on the invoice. Compute for sessions of about a minute was billed at $0.00.
• DigitalOcean's docs do not say whether unused plan balance rolls over. Check that before you subscribe.

Prerequisites

• A DigitalOcean account. Nothing here needs a plan; we did not subscribe to one.
• Node 20 or later, to run the calculator from the companion repo.
• Optional: an API token with billing read access, if you want to price your own invoice with the same scripts.

What you actually buy

Managed Agents runs coding agents such as Claude Code, Codex and OpenCode in microVMs, and bills for the parts separately: compute, memory, storage, tool calls, egress and model tokens. Agent Droplets does not change any of that. It adds a monthly fee that funds a balance, and your usage draws from that balance at a discount.

The figures come from DigitalOcean's Harness Runtime pricing page and the Agent Droplets pricing page, both read on 10 October 2026.

Not everything is discounted. The docs list what is:

• Harness Runtime compute (CPU) and memory
• Session storage, snapshots and custom templates
• Inference on DigitalOcean-hosted models, such as Kimi K3 and GLM-5.3
• Action Gateway tools: code interpreter, browser automation, web search and web fetch
• Public internet egress

Third-party models such as Anthropic Claude and OpenAI GPT are billed at standard rates and do not draw down the plan balance at a discount. So if your agents run on Claude, the plan saves you nothing on the biggest part of the bill.

When the balance runs out, nothing stops. In DigitalOcean's words, "you just move from the discounted rate to list price for the rest of the month."

Warning

We could not find what happens to unused balance at the end of the month. Neither pricing page says whether it rolls over. If it expires, a quiet month costs the full fee for less than the full amount of usage. Ask DigitalOcean before you subscribe to Team.

The math

Write your monthly discount-eligible usage at list price as U: compute, memory, storage, tools, egress and DigitalOcean-hosted models. Claude, GPT and other charges the plans do not discount are added to every line unchanged, so leave them out of U. Then:

Pay as you go:  U
Pro:            50  + max(0, U - 58.82)
Team:           200 + max(0, U - 250)

Pro costs a flat $50 until your usage passes $58.82, then grows with usage again. So Pro is cheaper than pay as you go as soon as U is above $50, and the gap never grows past $58.82 - $50 = $8.82. Team overtakes Pro where Pro's cost reaches $200, at U = $208.82, and its saving against pay as you go is capped at $250 - $200 = $50.

Monthly cost by plan

Monthly cost against monthly Managed Agents usage at list price. Only usage that the plans discount counts here; Claude and GPT tokens do not. Plan terms read 2026-10-10.

The shape is the whole story. Below $50, Pro costs more than it saves. Between $50 and $208.82, Pro is the cheapest, by at most $8.82. Above that, Team is cheapest, by at most $50 against pay as you go. A plan is worth it only once you know that your usage sits reliably above those lines.

The useful question is what $50 of usage looks like in agent runs. For that, we needed the real cost of a real run.

What our agent runs actually cost

Last month we built an issue-to-pull-request agent on Managed Agents: OpenCode on DeepSeek V4 Pro in a 2 vCPU / 4 GB microVM, one session per GitHub issue. That post estimated the model cost of each run from the token counts that the agent reported. The September invoice has now arrived, and it tells a different story.

The invoice API returns one line per session for compute, and one line per session for each kind of token: input, output and cache read. Stage 1 of the companion repo pulls those lines, and stage 2 prices them.

The-DevOps-Daily/agent-droplets-break-even on GitHub

These are the five recorded runs, matched to their invoice sessions by the session name in each run's log:

Two things stand out.

Cache-read tokens are most of the volume. An agent sends its growing conversation to the model on every step, and the provider bills the repeated part as cache reads, at one fifth of the input price for this model. Our earlier post counted only input and output tokens, so it put the model cost at about two cents per run. With cache reads and the invoice's rounding, the real figure was five to eight cents. We have added a correction to that post.

Each line was rounded to the cent, with a one-cent minimum. We checked a rule against all 31 token lines on our September invoice: billed = max($0.01, exact amount rounded to the cent). All 31 match. That is a pattern on one pay-as-you-go invoice, not a published billing policy, but it is consistent. On a long session this hardly matters. On short agent runs it does. For our five runs it added 17%, and across all 13 sessions on the invoice, including short probes, it added 25%: $0.62 billed for $0.497 of tokens. A probe that produced four output tokens was billed a full cent for them.

Exact token cost vs billed, per run

DeepSeek V4 Pro at $1.74 / $3.48 / $0.348 per million input / output / cache-read tokens. Billed amounts are the invoice lines before account credits, September 2026.

Compute was billed at $0.00. Compute is billed per second for the CPU a session actually uses, plus its peak memory. Even at full use, 2 vCPU and 4 GB list at $0.126 an hour, and the five sessions existed for 38 to 79 seconds each, so each one's compute was less than three tenths of a cent. The invoice shows $0.00 for every compute line. Unlike the token lines, compute lines below a cent were not raised to a cent.

So one recorded run cost $0.064 on the bill, tokens and compute together. Two of the five runs did not end in a merged pull request, so a merged pull request cost $0.107.

Turning $50 into runs

Stage 3 finds the first monthly volume at which each plan is cheaper. These are estimates: they assume a plan draws down its balance at the same charges our pay-as-you-go invoice showed, which we could not check without subscribing.

break-even

$ node scripts/03-break-even.mjs data/priced-2026-09.json
Pro is cheaper than pay as you go above $50.00 a month of discount-eligible usage at list price; it saves at most $8.82.
Team is cheaper than Pro above $208.82; it saves at most $50.

workload                       cost per unit  Pro cheaper from      Team cheaper than Pro from
issue-to-PR run (2026-09)      $0.0640        782 runs/month        3263 runs/month
merged pull request (2026-09)  $0.1067        469 merged PRs/month  1958 merged PRs/month

782 runs a month is about 26 runs a day, every day. A team that points an agent at every new issue in a busy repository might get there. A team that uses it a few times a day will not, and pays $50 for less than $50 of usage.

One long session: $0.78 for 39 minutes

Short runs are the worst case for the rounding rule. To see the other end, stage 4 of the repo ran one longer session the way a person works with a coding agent: six prompts, one after another, in the same 2 vCPU / 4 GB session, on GLM-5.3, a DigitalOcean-hosted model that the plans discount. The prompts asked the agent to read a small Python package, add ISO 8601 duration parsing, add a command-line entry point, rewrite the error messages, add type hints and documentation, and then review its own changes. The session ran for 39 minutes, changed five files with 346 added lines, and finished with 45 tests passing. Nothing was pushed.

It appeared on the invoice preview the next morning:

long session

$ node scripts/01-invoice-lines.mjs preview
data/invoice-2026-10.json: 4 Managed Agents lines { 'Harness Runtime Serverless Inference': 0.74, 'OHS Compute': 0.04 }
$ node scripts/02-price-sessions.mjs data/invoice-2026-10.json
all sessions: { exactModelCost: 0.7417, billedModelCost: 0.74 } invoice says: { 'Harness Runtime Serverless Inference': 0.74, 'OHS Compute': 0.04 }
rounding rule: {
  rule: 'max(0.01, round(exact, 2)) per token line',
  lines: 3,
  matches: 3
}

What the 39-minute session was billed

Invoice preview lines for one GLM-5.3 session in a 2 vCPU / 4 GB microVM, October 2026. List prices on the invoice: $1.40 input, $4.40 output and $0.26 cache read per million tokens.

Three things changed from the short runs:

• The rounding rule stops mattering. The three token lines match the same rule, but on $0.74 of tokens it moved the bill by less than a cent.
• Cache reads are a third of the bill. Each prompt sends the whole conversation so far. The cache makes that cheap per token, but the session read 971,520 cached tokens, 37 times its 26,087 fresh input tokens, and paid $0.25 for them. A long session pays for its own history.
• Compute is still small. $0.04 for 2,318 seconds. At full use, 2 vCPU and 4 GB would list at about $0.08 for that time, so the session was billed for roughly half, which fits billing for the CPU actually used. Tokens were 95% of the bill.

At $0.78 a session, the break-even comes much sooner:

calc

$ node calc.mjs --runs 65 --cost 0.78
Discount-eligible usage at list price: $50.70 a month
  Pay as you go  $50.70
  Pro            $50.00
  Team           $200.00
Cheapest: Pro.
$ node calc.mjs --runs 268 --cost 0.78
Discount-eligible usage at list price: $209.04 a month
  Pay as you go  $209.04
  Pro            $200.22
  Team           $200.00
Cheapest: Team.

Pro is cheaper from 65 sessions like this a month, about three every working day for one person. Team is cheaper than Pro from 268, which is about 13 people each running one such session every working day. This is one session on one model, so treat $0.78 as an example, not a rate. A longer conversation reads more cache, and the same token counts on Kimi K3 would list at about $1.91, 2.6 times as much.

Where the plan makes sense

• Agents that run for hours. Long interactive sessions use far more tokens than our short runs, and a busy agent keeps the CPU working. Our one recorded 39-minute session cost 12 times as much as a short run, and sessions like it reach the $50 and $208.82 lines much faster.
• Agents on DigitalOcean-hosted models. GLM-5.3 at $1.40 / $4.40 per million input / output tokens and Kimi K3 at $3 / $15 are discounted. Claude Sonnet 5.5 at $2 / $10 is not, even though it is billed through the same account.
• Many people sharing one account. Every plan includes unlimited seats and agents, so the whole team's usage counts toward one balance. With short runs like ours, ten people would need about 11 runs each per day to pass $208.82; with long sessions, far fewer.

Where it does not

• Occasional or bursty use. Below $50 a month, pay as you go is cheaper, and the free trial's $5 credit covers a lot of short runs.
• Claude Code or Codex on their own vendors' models. The model tokens are most of the bill, and the plan does not touch them.
• Before you have a month of real usage. Run pay as you go for a month, then price the invoice. The scripts do that for you.

Price your own usage

Clone the repo and pass your own numbers:

git clone https://github.com/The-DevOps-Daily/agent-droplets-break-even
cd agent-droplets-break-even
node calc.mjs --runs 1500 --cost 0.064

calc

$ node calc.mjs --runs 1500 --cost 0.064
Discount-eligible usage at list price: $96.00 a month
  Pay as you go  $96.00
  Pro            $87.18
  Team           $200.00
Cheapest: Pro.

To price your own invoice instead, run stage 1 with a billing-read token and your invoice id (or preview for the current month), then stage 2 on the file it writes. Stage 2 also checks the rounding rule against every token line, so you can see whether your bill follows it too.

What we could not conclude

• How a plan invoice looks. We stayed on pay as you go. We do not know whether the 15% discount is applied before or after the one-cent rounding, which matters for short runs.
• Rollover. As above, the docs do not say.
• A typical long session. We priced one. Its cost depends on the model, the length of the conversation and how much the agent reads back, so your sessions can cost a lot more or less than $0.78.
• Speed. Managed Agents is in preview, and DigitalOcean's preview terms treat benchmark results as confidential, so this post prices usage only.
• Prices after the preview. Plan terms and list prices can change before general availability. The repo keeps every price in one file with its source and the date we read it.

Summary

Agent Droplets is a commitment discount, not a cheaper server. It pays off only once your discounted usage reliably passes $50 a month, and even then it saves at most $8.82 on Pro and $50 on Team. Our short issue-to-PR runs cost six cents each on the bill, so a team would need about 780 of them a month before Pro is worth it. Sessions like our recorded long one reach it much sooner: that 39-minute session on GLM-5.3 cost $0.78, so about 65 of them a month pass the Pro line. Whichever you run, read your invoice before you subscribe: it showed us cache-read tokens and a rounding rule that the token counts alone did not.

The first change hits anyone who logs in to Windows or AD-joined machines as CORP\alice. Microsoft's own OpenSSH guide shows exactly that form: ssh domain\username@servername. The second change hits anyone who turned on -C for a slow link, and Ansible users on its default OpenSSH connection, because Ansible passes -C by default. We built OpenSSH 10.5p1 and 10.6p1 from source on the same machine and measured what each change does.

TLDR

• With a 10.6 client, CORP\alice and svc-backup$ were refused before any connection on 11 of the 13 ways we passed them: user@host, -l, -o User=, ssh:// URIs (also percent-encoded), scp, sftp, rsync, git, Ansible's ansible_user and a User ${VAR} line in ssh_config. With 10.5, all 26 attempts reached the server.
• Two ways still work: a literal User CORP\alice line in ssh_config, and the alice@corp.example.com@host form.
• ssh -C on 10.6 compressed a 32 MiB access log to 21.09 MiB. 10.5 compressed the same file to 3.71 MiB. That is 5.7 times more compressed data to send. JSON logs and source code came out 3.1 to 3.4 times bigger.
• The side that sends the data decides. If you upgrade only your laptop, downloads from an old server still compress well, and uploads do not.
• zstd -1 in a pipe compressed the access log 7.6x in 0.44 CPU seconds. zlib level 6, which ssh -C used before 10.6, needed 1.50 CPU seconds for 8.7x.
• A default 10.6 sshd logged a warning for paramiko 5.0.0, libssh2 1.10.0 and any client limited to classic key exchange. asyncssh 2.24.1, Go's x/crypto/ssh v0.58.0 and OpenSSH 9.2p1, 10.5p1 and 10.6p1 on default settings caused no warning. The setting is global only, so you cannot turn it off for one subnet.

Prerequisites

• A shell with OpenSSH. Run ssh -V to see your client version.
• Optional: the demo repo, which builds both versions and runs every test in this post. It needs a C toolchain, libssl-dev and zlib1g-dev, and builds both versions from source.

The-DevOps-Daily/openssh-10-6-check on GitHub

How we tested

Everything ran on one Debian 12 machine (aarch64, 4 cores, recorded in results/00-environment.txt). 01-build.sh downloads openssh-10.5p1 and openssh-10.6p1, checks the SHA256 sums printed in each release note, and builds both into separate prefixes. Each sshd runs as an ordinary user on 127.0.0.1. The username test uses one 10.6 sshd and points both client versions at it. The server log is the evidence: if sshd logs Invalid user CORP\\alice, the name got through. If the client prints remote username contains invalid characters and the log has no new line, the client refused it before connecting.

All usernames other than the local account are fake, so each login that reaches the server ends in Permission denied. That is the result we want here. It proves the client sent the name.

Change 1: usernames with a backslash or a dollar sign

The new check is in ssh_valid_ruser() in readconf.c. The list of forbidden characters gained two entries, $ and \. The release notes say why: a username from an untrusted source can end up in a shell through %r in ProxyCommand or Match exec. The check runs on every username that came from the command line, and on any User value from ssh_config that changed when OpenSSH expanded it.

Here is the same command on both versions, recorded by 09-transcript.sh:

OpenSSH 10.5p1 and 10.6p1, same command

# every ssh below also gets -F work/base_config: port 18022, the test key, BatchMode
$ ssh 'CORP\alice@127.0.0.1' true   # 10.5p1
CORP\\alice@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).
$ ssh 'CORP\alice@127.0.0.1' true   # 10.6p1
remote username contains invalid characters
$ ssh -o 'User=CORP\alice' 127.0.0.1 true   # 10.6p1
remote username contains invalid characters
$ head -3 ssh_config   # the Host block the next two commands use
Host winbox
  HostName 127.0.0.1
  User CORP\alice
$ ssh winbox true   # 10.6p1, User line in ssh_config
CORP\\alice@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).
$ WIN_USER='CORP\alice' ssh winbox true   # 10.6p1, User ${WIN_USER} in ssh_config
remote username contains invalid characters
$ ssh 'alice@corp.example.com@127.0.0.1' true   # 10.6p1, UPN form
alice@corp.example.com@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).

The doubled backslash in CORP\\alice is how OpenSSH prints the name. The server received CORP\alice.

03-usernames.sh runs four usernames through thirteen entry paths with each client version. This is the 10.6p1 column. With 10.5p1, every CORP\alice and svc-backup$ cell reached the server.

scp, sftp, rsync and git all start ssh with the username as an argument, so they get the same refusal. The message does not change. git prints it and exits with 128. Ansible wraps it:

t | UNREACHABLE!: Task failed: Failed to connect to the host via ssh: remote username contains invalid characters

The Ansible row matters because the usual inventory for Windows over SSH sets ansible_user: 'CORP\alice'. ansible-core 2.21.5 passes that as -o User=, which counts as the command line.

Who this breaks

• Admins on macOS (Homebrew), Arch or Debian unstable who log in to Windows Server with DOMAIN\user, as in Microsoft's guide.
• Linux hosts joined to Active Directory through SSSD, where people log in with the short domain form.
• Ansible inventories that manage Windows over SSH with a down-level logon name in ansible_user.
• Scripts that build ssh "$user@$host" from a variable that can contain \ or $.

Windows ships its own OpenSSH build on its own schedule. We did not test the Windows client.

What to do

1. Use the user@domain form. The Win32-OpenSSH examples list ssh user@domain@host next to ssh domain\user@host. OpenSSH splits at the last @, so the name reaches the server intact. Inside an ssh:// URI, write it percent-encoded as alice%40corp.example.com. We only checked that the name reaches the server. Whether your Windows server or SSSD setup accepts that form is up to its configuration, and we did not test it.

2. Put the name in a literal User line in ssh_config. The check skips literal values:

   Host win-*.corp.example.com
     User CORP\alice
   

3. For Ansible, remove ansible_user from those hosts and let ssh_config supply the name. Pass the file with ansible_ssh_common_args: '-F /path/to/ssh_config', or put the block in ~/.ssh/config. Our test shows this path reaches the server on 10.6.

4. Do not template the username into ssh_config as User ${VAR}. 10.6 checks the expanded value and refuses it.

Change 2: ssh -C still compresses, but much less

Before 10.6, packet.c started the compressor with deflateInit(stream, 6), which is zlib level 6, the same as gzip -6. In 10.6 the call is deflateInit2(..., Z_BEST_SPEED, Z_DEFLATED, 15, 8, Z_HUFFMAN_ONLY). Huffman-only deflate still gives short codes to common bytes. It does not replace repeated strings with references to earlier data, and that is where text gets most of its compression.

The reason is a 2026 paper, "Crossing the Streams". All channels in one SSH session share one compression dictionary. If an attacker can send data through one channel, such as a forwarded port, the size of the compressed output can leak secrets from another channel: when the attacker's guess matches the secret, deflate replaces it with a short back-reference and the packet gets smaller. Huffman-only coding has no back-references, which removes that signal. The output size still depends a little on which bytes are sent, so the ssh_config manual in 10.6 still says not to enable compression on connections that mix trusted and untrusted traffic.

05-compression.sh sends four files through every client and server pair, up and down, three times each. ssh -v prints the raw and compressed byte counts at exit. These are the compressor's own counters, so SSH framing, MAC tags and TCP add a little on top. The counts changed by at most one byte between runs.

Compressed SSH payload with ssh -C, by the version of the side that sends

OpenSSH 10.5p1 and 10.6p1, zlib@openssh.com, median of 3 runs. access.log and events.ndjson are 32 MiB each, the OpenSSH source tar is 11.2 MiB. A 16 MiB random file came out at 16.01 MiB with both versions.

The access log, the JSON events and the random file are generated from fixed seeds, with IP addresses from the documentation ranges. The source tar is the real openssh-10.6p1 release, so it is the one file that we did not generate. As a cross-check, 07-app-compression.sh runs plain zlib with Z_HUFFMAN_ONLY on the same files. It gave 1.52x, 1.59x and 1.62x, the same as the 10.6 ssh results.

The sender decides

Each side of an SSH connection compresses what it sends. In our matrix, the upload ratio followed the client version and the download ratio followed the server version, in every case:

• A 10.6 client downloading from a 10.5 server got 8.62x on the access log.
• A 10.5 client downloading from a 10.6 server got 1.52x.

So the change reaches you in stages. Upgrade your laptop and your uploads get bigger. Your downloads change when the servers upgrade.

What it costs in time

Without a bandwidth limit, on loopback, the times in 05-compression.txt were mixed. The access log came back faster from a 10.6 server (1.84 s median, against 2.19 to 2.54 s from 10.5). The JSON file was slower in one pair (3.35 s against 2.47 s). The runs were not interleaved and the machine was shared, so we do not read CPU cost from them. On a slow link, the bytes decide.

08-shaped-link.sh adds a 20 Mbit/s tc rule on the loopback device, for one port only, and downloads the access log five times with a 10.6 client:

The slowest 10.5 run was faster than the fastest 10.6 run. The ranges are wide because the test machine was doing other work. The byte counts above are exact, so plan with them: at 20 Mbit/s, 3.71 MiB needs at least 1.6 s and 21.09 MiB needs at least 8.8 s.

Who this affects

• Anyone with Compression yes in ssh_config, or ssh -C and scp -C in scripts, over VPNs, satellite or mobile links.
• Ansible on its default OpenSSH connection. The default ssh_args in ansible-core 2.21.5 are -C -o ControlMaster=auto -o ControlPersist=60s, and we saw them in its -vvv output. Most of that traffic is small modules, but copy and fetch move whole files through the same connection.
• Probably less so rsync -z and git. Both compress their bulk data before SSH sees it, so -C should have had little left to do. We did not measure them.

What to do

Compress at the application level, as the release notes recommend. On our files:

# A file, compressed with zstd on the way and decompressed on arrival.
# -o Compression=no stops a Compression yes in ssh_config from compressing twice.
zstd -1 -c access.log | ssh -o Compression=no host 'zstd -dc > access.log'

# A directory
tar -cf - ./logs | zstd -1 | ssh -o Compression=no host 'zstd -dc | tar -xf - -C /srv/archive'

# rsync with its own compression (rsync 3.2 and newer can use zstd)
rsync -az --compress-choice=zstd ./logs/ host:/srv/archive/

Compressing one stream on its own, as above, keeps data from other channels out of its dictionary, so the attack from the paper does not apply to it. For data that is already compressed, such as images, archives and most backups, -C usually gains little, as the random file above shows, and you can leave it off.

Change 3: sshd now warns about classic key exchange

WarnWeakCrypto came to the client in 10.1. In 10.6 it is also a server option, on by default. Each connection that does not use a post-quantum key exchange adds one line to the log at INFO, the default level:

WARNING: connection from 127.0.0.1 port N is not using a post-quantum key exchange algorithm: "curve25519-sha256" [preauth]

06-weakcrypto.sh connects eight clients to a 10.5 and a 10.6 sshd, both with default settings, and reads the negotiated key exchange from the log:

The 10.5 sshd logged no warning for any of them. We did not test Fabric or Netmiko. Both are built on paramiko, so expect one warning per connection from them too, unless your version negotiates a post-quantum key exchange. If you alert on WARNING in auth logs, expect new alerts from automation hosts.

You can turn it off with WarnWeakCrypto no-pq-kex (only this warning) or WarnWeakCrypto no (all of them). The option is global only. In a Match Address block, sshd -t fails:

sshd-match-test.conf line 10: Directive 'WarnWeakCrypto' is not allowed within a Match block

So you cannot keep the warning for people and hide it for one automation subnet. The better fix is on the client side: update the library, or move the job to a client that supports mlkem768x25519-sha256.

What we could not conclude

• We tested one machine and one OpenSSL build (3.0.22). The username check and the compression change are in OpenSSH's own code, so other platforms should behave the same, but we only measured this one.
• We did not test the OpenSSH client that ships with Windows, or any vendor build that patches these changes out.
• The timings vary because the machine was shared. The byte counts and the refusals do not depend on load.
• Three of the four files are generated. The real source tar compressed less than either generated log (4.99x against 5.34x and 8.62x), and your own files will compress differently again.
• We checked that alice@corp.example.com reaches the server. We did not test whether a Windows server or an SSSD host accepts it for login.
• We tested libssh2 1.10.0 only, the version in Debian 12. Newer libssh2 releases may negotiate something else.

Summary

OpenSSH 10.6 makes three changes that people will see within days of upgrading. Logins written as CORP\alice fail with remote username contains invalid characters, through ssh, scp, sftp, rsync, git and Ansible, unless the name is a literal User line in ssh_config or you switch to alice@corp.example.com@host. ssh -C now compresses text about 1.5x instead of 5x to 9x, and the side that sends the data decides, so move the compression to zstd or rsync. And a 10.6 sshd logs one warning per connection for paramiko and older libssh2 clients, with no way to scope that per subnet. Run ssh -V on your laptop today. If it says 10.6, check your inventories and scripts for \ in usernames before the next on-call shift finds them for you.

"Mostly" is not "only". We wanted to know exactly what changes on Linux, so we ran the same 13 cases on Python 3.14.8 and 3.15.0, each with its default settings and with PYTHONUTF8=0, in seven locales, from a bare container to a Latin-1 server. A file read with no encoding is the least of it. The changes we found are in stdin and stdout, on hosts with an old locale, and in what the opt-out switch really gives you.

TLDR

• Container locale settings: no change. With no locale, LANG=C or LC_ALL=C, 3.14 and 3.15 gave the same result in all 13 cases. Python 3.7 and later already turn UTF-8 mode on in the C locale. The official python images set no LANG from 3.13, and LANG=C.UTF-8 before that. Both settings gave zero differences.
• UTF-8 servers (en_US.UTF-8): only stdin and stdout change. open(), subprocess and environment variables gave the same results. Bytes that are not valid UTF-8 on stdin crashed 3.14 and pass through on 3.15 as lone surrogates. When we turned them into JSON, jq and Node replaced them with U+FFFD, so the original byte was lost.
• Legacy locales (LANG=en_US, which is ISO-8859-1 in glibc, and ja_JP.eucJP): 11 and 10 of 13 cases change. UTF-8 data starts to work. ISO-8859-1 data stops working. In the ISO-8859-1 locale, 3.14 never raised an error on UTF-8 input. It gave mojibake.
• The opt-out is not "what 3.14 did". PYTHONUTF8=0 gave the same result on 3.14 and 3.15 in every case. What changed is the default: 3.14 turned UTF-8 mode on by itself in the C locale. So with LC_ALL=C, 3.15 with PYTHONUTF8=0 fell back to ASCII. 7 cases differed from 3.14's defaults, and 4 that worked raised errors.
• You can find the affected calls today. -X warn_default_encoding reported the same 4 calls on 3.14 and 3.15. On a sample script, our static checker and the runtime warning reported the same 6 lines.

Prerequisites

• Python 3.14 or 3.15 on Linux, and a shell
• Optional: the demo repo, which needs uv and glibc's localedef

What UTF-8 mode changes, and where it was already on

UTF-8 mode is not new. PEP 540 added it in Python 3.7. When it is on, Python ignores the locale encoding for four things, which the 3.15 docs list:

1. sys.getfilesystemencoding() returns utf-8, so file names, os.environ and sys.argv decode as UTF-8.
2. locale.getpreferredencoding() returns utf-8, so open(), Path.read_text() and subprocess with text=True use UTF-8 when you give no encoding.
3. stdin and stdout use UTF-8 with the surrogateescape error handler.
4. os.device_encoding() returns utf-8 for a terminal on Unix.

Before 3.15, the 3.14 docs say, it was on "if the LC_CTYPE locale is C or POSIX at Python startup". PEP 538, also in 3.7, coerces a C locale to C.UTF-8 and gives the coercion target locales surrogateescape on stdin and stdout too. That covers most containers. So on Linux, 3.15 changes behaviour in two places: real UTF-8 locales such as en_US.UTF-8, and locales that are not UTF-8 at all.

The test

The repo has 13 small cases. Each one does one thing that depends on the default encoding: read an ISO-8859-1 file, read a UTF-8 file, write café 25€, read a child process's output with text=True, read an ISO-8859-1 value from os.environ and sys.argv, read stdin, print a file name that is not valid UTF-8, and so on. Two of the 13 are controls that should not depend on the locale: a read with encoding="locale", and the EncodingWarning check. A separate check, handoff, has one version write a file and the other read it on the same host.

The runner starts every case from an empty environment, with only PATH, LOCPATH, PYTHONDONTWRITEBYTECODE and the locale variables. It runs each case with four interpreter settings (3.14.8 and 3.15.0, each with its default and with PYTHONUTF8=0) in seven locales:

localedef builds the locales into the repo, so the result does not depend on what the host has installed. Before it trusts a row, the runner checks that the locale reports the encoding it should. If it does not, the runner stops, so a missing locale cannot pass as a result. A case "differs" between two settings when anything recorded differs: the returned value, the error message, the exit code or the bytes written. We wrote the claim in CLAIM.md before the first run, with the results that would prove it wrong, and check_claim.py checks it against the recorded results.

python-315-utf8-default

$ ./scripts/setup.sh
locales: C.UTF-8 en_US en_US.UTF-8 ja_JP.eucJP
$ python3 scripts/run_matrix.py
406 results from 420 interpreter runs written to results/matrix.json

Then python3 scripts/check_claim.py checks the claim. This is what it wrote to results/claim-check.txt:

PASS  no locale: 3.14 and 3.15 agree in every case
PASS  LANG=C: 3.14 and 3.15 agree in every case
PASS  LC_ALL=C: 3.14 and 3.15 agree in every case
PASS  LANG=C.UTF-8: open(), subprocess and environment cases agree
FAIL  LANG=C.UTF-8: invalid UTF-8 on stdin/stdout differs between 3.14 and 3.15
      differs: nothing
PASS  LANG=en_US.UTF-8: open(), subprocess and environment cases agree
      stdin_latin1, stdin_to_json, stdout_filename
PASS  LANG=en_US.UTF-8: invalid UTF-8 on stdin/stdout differs between 3.14 and 3.15
      differs: stdin_latin1, stdin_to_json, stdout_filename
PASS  LANG=en_US: every case that uses the default encoding differs (11 of 13)
      agree: read_latin1_locale, encoding_warning
FAIL  LANG=ja_JP.eucJP: every case that uses the default encoding differs (10 of 13)
      agree: read_latin1_locale, environ_argv, encoding_warning
FAIL  3.15 + PYTHONUTF8=0 gives the 3.14 default result in every case
      LC_ALL=C: read_latin1; LC_ALL=C: read_utf8; LC_ALL=C: write_text; LC_ALL=C: subprocess_latin1; LC_ALL=C: subprocess_utf8; LC_ALL=C: stdin_utf8; LC_ALL=C: stdout_utf8
PASS  PYTHONUTF8=0 gives the same result on 3.14 and 3.15 in every case
PASS  -X warn_default_encoding reports the same 4 calls on 3.14 and 3.15

The three FAIL lines are where our claim was wrong. We expected C.UTF-8 to behave like en_US.UTF-8. We expected every default-encoding case to change in ja_JP.eucJP. And we expected PYTHONUTF8=0 to bring back 3.14 behaviour everywhere. All three are covered below.

Cases where 3.15 gives a different result from 3.14

13 cases per locale, Python 3.14.8 vs 3.15.0, Linux aarch64, glibc 2.36. A case differs when the value, error, exit code or bytes written differ. The last row compares 3.14 defaults with 3.15 run with PYTHONUTF8=0. Source: results/matrix.json in the demo repo.

Container locale settings: nothing changed

In the first three locales, 3.14 and 3.15 agreed in every case, including the ones that fail. An ISO-8859-1 file read with no encoding raised UnicodeDecodeError on both. A UTF-8 file read correctly on both. That is UTF-8 mode working on 3.14 already, because the locale was C.

We expected LANG=C.UTF-8 to show the stdin and stdout change, since 3.14 has UTF-8 mode off there. It did not. 3.14 already uses surrogateescape on stdin and stdout in C.UTF-8, because that is one of the locales PEP 538 coerces C to, and the PEP gives those locales the same error handler. So all four container-style locales agreed on all 13 cases.

That matters for images. The official python Dockerfiles dropped ENV LANG C.UTF-8 for 3.13 and later in December 2023, and the 3.12 and older images still set it. Both settings are rows with zero differences. We did not run the images themselves. We ran their locale settings with uv's Python builds on one glibc host, so treat this as the result for those settings, not a test of every image. Alpine uses musl, a different C library, and we did not test it.

UTF-8 servers: stdin and stdout are what change

In en_US.UTF-8, the open(), subprocess and environment cases agreed. The locale encoding was UTF-8, so the default was UTF-8 before and after. Three cases changed, and all three are about bytes that are not valid UTF-8 on stdin or stdout:

3.14 used the strict error handler on stdin and stdout in a normal UTF-8 locale. 3.15 uses surrogateescape, which maps each bad byte to a lone surrogate (\udce9 for 0xe9). You get the original byte back only where something encodes the text with surrogateescape again. stdout does, which is why printing the odd file name now works. A strict UTF-8 encode, such as writing to a file opened with encoding="utf-8", raises. And some consumers replace the surrogate. So the error moves from the input to a later step, or there is no error and the data changes.

The JSON case shows the second outcome. json.dumps with its defaults writes the surrogate as the escape \udce9, which is valid JSON syntax but not a valid character. RFC 8259 section 8.2 says the behaviour of software that receives such values "is unpredictable". We piped the line into two consumers:

input bytes on stdin:   63 61 66 e9 0a (the script strips the newline)
3.15 wrote:             {"line": "caf\udce9"}
jq-1.6 -r .line: 63 61 66 ef bf bd 0a
node v24.14.0 JSON.parse, written as UTF-8: 63 61 66 ef bf bd 0a
ef bf bd is U+FFFD, the replacement character: the original byte e9 is gone.

On 3.14 this pipeline crashed at the read, on a UTF-8 server. On 3.15 it runs, and both consumers we tried replaced the byte with U+FFFD. In the container locales, 3.14 already wrote the same line, so the risk is not new. What is new is that a log shipper or ETL script that crashed on bad input on your UTF-8 servers now writes slightly wrong data instead.

If you want the 3.14 crash back for one script, either of the first two lines below works on 3.15. The third keeps the raw bytes instead:

default:                      'caf\udce9\n'
stdin.reconfigure(strict):    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte
PYTHONIOENCODING=utf-8:strict UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte
sys.stdin.buffer (bytes):     b'caf\xe9\n'

sys.stdin.reconfigure(errors="strict") at the top of the script is the most explicit way to fail at the input. If the input can be any bytes, read sys.stdin.buffer and decode it yourself with the encoding you expect.

Legacy locales: one set of failures for another

On a host with LANG=en_US, glibc uses ISO-8859-1. 11 of the 13 cases changed, which is every case except the two controls:

• Five got better. Reading a UTF-8 file, writing café 25€, reading UTF-8 from a child process, reading UTF-8 on stdin, and printing café 25€ all work on 3.15. On 3.14, the two that write € raised errors, because ISO-8859-1 has no €. The other three gave mojibake with no error: 'caf\xc3\xa9 25\xe2\x82\xac' instead of café 25€. ISO-8859-1 maps every byte to a character, so decoding never fails. It only gives wrong text.
• Five got worse. Reading an ISO-8859-1 file now raises UnicodeDecodeError. So does reading ISO-8859-1 output from a child process. An ISO-8859-1 value in os.environ or sys.argv now decodes as 'caf\udce9' and fails later, when you encode it as UTF-8. Stdin turns into surrogates, and the JSON line changes from caf\u00e9 to caf\udce9.
• One printed the same bytes by a different route: os.listdir() returned a different string, and stdout turned it back into the same bytes.

In ja_JP.eucJP, 10 of 13 cases changed. The environment case did not, because its ISO-8859-1 byte is invalid in EUC-JP too, so both versions decode it to the same surrogate. Here 3.14's EUC-JP decoder raised on UTF-8 input, where ISO-8859-1 had given mojibake. Our fixtures hold UTF-8 and ISO-8859-1 text only, so this row says nothing about valid EUC-JP data. If you have Japanese legacy data, test it with your own files.

The handoff check shows the risk during a rolling upgrade. On the LANG=en_US host, a 3.15 job wrote café 25€ as UTF-8, and a 3.14 job read it back as 'caf\xc3\xa9 25\xe2\x82\xac\n', with no error. The other direction failed loudly: 3.14 could not encode € in ISO-8859-1, and because open("w") creates or truncates the file before the write, it left an empty file.

If your legacy data really is ISO-8859-1, say so in the code. encoding="latin-1" reads it correctly on every version. encoding="locale", added in Python 3.10, uses the locale encoding even in UTF-8 mode. It read the ISO-8859-1 file correctly on 3.14 and 3.15 in the en_US locale.

The opt-out is not "what 3.14 did"

The release notes give one way back: "To retain the previous behaviour, Python's UTF-8 mode may be disabled with the PYTHONUTF8=0 environment variable or the -X utf8=0 command-line option." The switch itself did not change. With PYTHONUTF8=0, 3.14 and 3.15 gave the same result in all 13 cases in all seven locales. What changed is the default, and the default in 3.14 was not "off". It was "on in the C locale, off elsewhere". PYTHONUTF8=0 is off everywhere.

In six of the seven locales, that made no difference: 3.15 with PYTHONUTF8=0 matched 3.14's defaults. With LANG=C, PEP 538 still coerces the locale to C.UTF-8, so the encoding stays UTF-8. With LC_ALL=C, coercion does not happen (PEP 538 skips it when LC_ALL is set), so the locale encoding is ASCII, and that is what 3.15 used:

## LC_ALL=C with PYTHONUTF8=0: reading a UTF-8 file
no encoding:                  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
encoding="utf-8":             'caf\xe9 25\u20ac\n'

7 of the 13 cases differed from 3.14's defaults. Four that worked on 3.14 raised: reading a UTF-8 file, writing café 25€, reading UTF-8 from a child process, and printing café 25€. One more turned UTF-8 on stdin into surrogates with no error. The other two failed on both, with a different error message.

So do not put PYTHONUTF8=0 in a base image or a shared profile as a "keep the old behaviour" switch. Any job that also runs with LC_ALL=C gets ASCII, which 3.14 with its defaults never gave it. If you need the opt-out, set it only for the processes that run in a legacy locale and read legacy data, and plan to remove it.

What to do before you upgrade

Most of this goes away when the code names its encoding. Three steps, in order:

1. Find the calls. The repo has a static checker, check_default_encoding.py. It reports open() without encoding (or with encoding=None), Path.read_text() and write_text(), .open() on a Path it can see, subprocess in text mode with no encoding, os.popen(), logging file handlers, tempfile and gzip.open() in text mode, io.TextIOWrapper() and locale.getpreferredencoding(). On a sample release-notes script, it found 6 calls:

   examples/release_notes.py:12:1: logging.basicConfig(filename=...) without encoding
   examples/release_notes.py:16:11: subprocess.run() in text mode without encoding
   examples/release_notes.py:23:24: .read_text() without encoding
   examples/release_notes.py:24:16: open() without encoding
   examples/release_notes.py:27:5: .write_text() without encoding
   examples/release_notes.py:28:10: open() without encoding
   6 call(s) use the default encoding
   

   When the same script ran on 3.15 with -X warn_default_encoding, Python reported EncodingWarning on the same 6 lines. A static check misses calls through variables, wrappers and **kwargs it cannot read, so do step 2 as well.

2. Run your tests with the warning as an error. This works on 3.10 and later, so you can do it on 3.14 now:

   python -X warn_default_encoding -W error::EncodingWarning -m pytest
   

   In our matrix, the warning fired for the same 4 calls on 3.14 and 3.15. Fix each open()-style call with encoding="utf-8", or encoding="locale" where the file really follows the locale. locale.getpreferredencoding() takes no encoding: replace it with locale.getencoding() if you need the locale encoding, or with "utf-8" if that is what the code meant.

3. Decide what stdin and stdout should do with bad bytes. If a script reads input it does not control, such as logs, file names or another tool's output, choose on purpose: sys.stdin.reconfigure(errors="strict") to fail at the input, or sys.stdin.buffer to keep the bytes. Do not let the default choose for you.

Then check which locale your hosts really run in. locale and python3 -c "import locale; print(locale.getencoding())" on each host type tell you. If every host is C, C.UTF-8 or no locale, none of our 13 cases changed for those settings.

The-DevOps-Daily/python-315-utf8-default on GitHub

What we could not conclude

• Windows was not tested. The PEP says Windows is where most of the change is, because the locale encoding there is the ANSI code page, for example cp1252. If you run Python on Windows hosts or build agents, run the checker and the warning there first.
• macOS and musl (Alpine) were not tested. We ran uv's Python builds on Linux aarch64 with glibc 2.36. We have no result for either platform.
• The EUC-JP row used UTF-8 and ISO-8859-1 fixtures, not valid EUC-JP text.
• 13 cases are not your code. Third-party libraries that call locale.getpreferredencoding() themselves now get utf-8 in UTF-8 mode. The checker only sees the code you point it at.
• We did not count how many hosts use legacy locales. We know the behaviour on such a host, not how common those hosts are.

Summary

PEP 686 is a smaller change for Linux containers than its headline suggests, because Python has used UTF-8 mode in the C locale since 3.7. With the locale settings that containers use, it changed none of our 13 cases. It changes how stdin and stdout treat invalid bytes on UTF-8 servers, so a crash at the input becomes a lone surrogate further down the pipeline. It changes almost everything on hosts with a legacy locale. And its opt-out, PYTHONUTF8=0, means "off", so LC_ALL=C processes get ASCII, which 3.14 with its defaults never gave them. Name the encoding in your code, run your tests once with -X warn_default_encoding, and decide what your scripts do with bytes that are not UTF-8.

So your laptop and your servers now store logs in two different formats. Most of the time you will not notice, because docker logs reads both. You will notice if something reads the log files directly, such as a collector that tails /var/lib/docker/containers/*/*.log. We wrote the same log lines through both drivers on Docker Engine 29.9.0 and recorded what each one stores, what docker logs gives back, and what a file-tailing collector reads.

TLDR

• Docker Desktop 4.94 uses local for new Linux containers. Docker Engine still defaults to json-file with max-size unlimited, and Docker's docs say that is only for backward compatibility.
• 500,000 log lines (47.9 MB of text) took 83.8 MB with json-file and no options. local kept all 500,000 lines in 24.1 MB. json-file capped at 10 MB x 3 used 23.8 MB, but kept only 142,146 lines.
• docker logs returned all 500,000 lines from the uncapped json-file container and from the local one.
• Fluent Bit's tail input, set up the usual way for a Docker host, read 500,000 records from the json-file container and 0 from the local one. Pointed at the local file directly, it read binary framing that the docker parser could not parse.
• A changed default only reaches containers created after the change. Existing containers keep json-file until you recreate them.

Prerequisites

• Docker Engine or Docker Desktop, and docker info access
• sudo on a Linux Docker host, if you want to look at the log files yourself
• Optional: the demo repo, which runs every test in this post

What changed, and what did not

Docker has always written container output to files on the host, in a format chosen by the log driver. The default has been json-file since the early days: one JSON object per line, with the text, the stream and a timestamp. Its documentation gives max-size a default of -1, which means unlimited, so one noisy container can fill the disk. The logging overview says Docker keeps json-file without rotation as the default "to remain backwards compatible with older versions of Docker", and recommends local for other cases because "it performs log-rotation by default, and uses a more efficient file format."

The local driver keeps 5 files of 20 MB each per container, so 100 MB of log text, and compresses the files it rotates out. Docker Desktop 4.94 now uses it for new Linux containers. Docker Engine 29.9.0, the newest Engine release when we tested, still reports json-file:

Docker Engine 29.9.0 on Linux

$ docker version --format 'docker client {{.Client.Version}}, engine {{.Server.Version}}'
docker client 29.9.0, engine 29.9.0
$ docker info --format '{{.LoggingDriver}}'
json-file

We could not run Docker Desktop on our test machine. The Desktop behaviour in this post comes from its release notes. Every measurement comes from Docker Engine 29.9.0 with the same two drivers. The output above is from a daemon with no logging settings (runs/.../00-default.txt).

The test

The demo repo starts three containers that each print the same 500,000 lines, about 96 bytes each, shaped like access logs:

• logs-json-default: json-file with no options, which is what Docker Engine does out of the box
• logs-json-capped: json-file with max-size=10m and max-file=3, a common hand-made fix
• logs-local: local with its defaults, which is what Docker Desktop 4.94 now does

Then it measures the files each driver wrote, counts what docker logs returns, and runs a real collector against the files. We ran it on a Raspberry Pi 4 (arm64) with Docker Engine 29.9.0 and Fluent Bit 5.1.3. The outputs below come from runs/2026-10-09-engine-29.9.0/ in the repo.

Same 500,000 lines, three setups (excerpt)

$ LINES=500000 ./scripts/01-write-logs.sh
logs-json-default: driver=json-file options={}
logs-json-capped: driver=json-file options={"max-file":"3","max-size":"10m"}
logs-local: driver=local options={}
$ ./scripts/02-measure.sh
container          driver       file_bytes  files   lines_kept  bytes_of_text
logs-json-default  json-file      83833258      1       500000       47888900
logs-json-capped   json-file      23833155      3       142146       13614435
logs-local         local          24134700      3       500000       47888900
# the file listings and byte dumps that follow are in 02-measure.txt

Size of the log files for the same 500,000 lines

Docker Engine 29.9.0, 500,000 lines of about 96 bytes (47.9 MB of text). json-file with no options kept all 500,000 lines, json-file capped at 10 MB x 3 kept 142,146, local kept all 500,000. These lines repeat a lot; compression depends on your log content.

Three things stand out:

1. json-file costs more than the text. 47.9 MB of text took 83.8 MB of files, 1.75 times the text, because every line is wrapped in {"log":...,"stream":...,"time":...}. That is about 72 extra bytes per line here. With no max-size, the file only grows.
2. Our size cap on json-file threw data away. The capped container used 23.8 MB and docker logs returned only 142,146 of the 500,000 lines. The rest rotated out.
3. local kept everything in about the same space. It used 24.1 MB and returned all 500,000 lines. The current file was 19.4 MB, uncompressed, framing included. Each of the two rotated files was about 2.3 MB after gzip, and the driver rotates a file at 20 MB before compression.

The two capped setups do not have the same budget. json-file at 10 MB x 3 allows 30 MB. local allows 5 files of 20 MB each before compression, framing included, and it also drops its oldest lines when it reaches that limit. A container that writes more than about 100 MB of log records loses lines with either driver. These lines repeat a lot, so they compressed well; how well your logs compress depends on their content.

The file format is what changes

docker logs returned all 500,000 lines from both uncapped containers, because it reads through the Docker daemon. The files on disk are a different story. Here are the first bytes of each file, from the same run:

First bytes of each driver's file, from 02-measure.txt (first 3 of 6 od lines)

# json-file: <container-id>/<container-id>-json.log
0000000   {   "   l   o   g   "   :   "   2   0   2   6   -   1   0   -
0000020   0   9   T   1   0   :   0   0   :   0   0   .   0   0   0   0
0000040   0   1   Z       l   e   v   e   l   =   i   n   f   o       m
# local: <container-id>/local-logs/container.log
0000000  \0  \0  \0   s  \n 006   s   t   d   o   u   t 020 205 330 232
0000020 342 244 221 265 356 030 032   _   2   0   2   6   -   1   0   -
0000040   0   9   T   1   0   :   0   0   :   0   0   .   3   4   1   6

json-file writes one JSON object per line, in a file named <id>-json.log in the container's directory. local writes length-prefixed binary records in a local-logs/ subdirectory, named container.log, and the rotated files end in .gz. Docker's docs say these files "are designed to be exclusively accessed by the Docker daemon", and that reading them with external tools "should be avoided".

What a file-tailing collector sees

Many Docker hosts ship logs with an agent that tails the files. We ran Fluent Bit 5.1.3 with its tail input and the docker parser on each container's *.log files, the usual setup for a Docker host:

Fluent Bit tail input, docker parser (excerpt)

$ ./scripts/03-collector.sh
Fluent Bit: Fluent Bit v5.1.3

== the usual Docker host pattern: <container dir>/*.log, docker parser
logs-json-default  records read: 500000
logs-json-capped   records read: 22860
logs-local         records read: 0

== aimed at the local driver's own file: local-logs/container.log
records read: 158382
records the docker parser could parse: 0
# two sample records follow; the first 20 are in 03-local-raw-head.txt

The json-file container gave all 500,000 records. For the local container the counter reported 0, because *.log in the container directory does not match local-logs/container.log. When we pointed Fluent Bit at that file directly, it read 158,382 records and the parser understood none of them. The tail input splits on newline bytes, which this binary format also uses inside its framing, so the records are fragments: the first one holds only a length byte, and the others hold one message each with binary framing around it, in the form ^Fstdout^P... (shown with cat -v). It also left out the two rotated .gz files.

The capped container's 22,860 is a side effect of the test. Fluent Bit started after the container had finished, so it found only the current -json.log file. A collector that runs all the time can read lines before they rotate, but this run did not test that.

What this means for other collectors depends on how they read:

• Collectors that read through the Docker daemon should keep working, because the daemon reads the local format for them, as it does for docker logs. Grafana Alloy's loki.source.docker "reads log entries from Docker containers" through the daemon address you give it. Vector's docker_logs source connects to docker_host. We did not test either of them with local.
• Collectors that parse the files as Docker JSON cannot decode local. Fluent Bit's tail input with the docker parser is one, as measured above. Filebeat's container input reads files and parses the docker and cri formats, and the local format is neither. That input is deprecated, and Filebeat's docs say to use the filestream input with its container parser instead.

We tested Fluent Bit only. For the others, the docs above are our source, so test your own agent before you depend on this.

Existing containers keep their driver

The log driver is set when a container is created. Docker's docs say that changing the default "only affects containers that are created after the configuration is changed". We checked it: we created a container, restarted the daemon with local as the default, then created a second one and restarted the first.

Changing the daemon default

$ ./scripts/04-default-change.sh before
== before: daemon default is json-file
logs-before: json-file
# restart dockerd with --log-driver local
$ ./scripts/04-default-change.sh after
== after: daemon default is local
logs-before (created before, started again now): json-file
logs-after  (created after): local

We tested this on Engine, not on a Desktop upgrade, but the rule is the same. On a laptop that you upgrade to 4.94, the containers you already have keep the driver they were created with. A container that you remove and run again, or that Compose recreates, gets the new default, unless its configuration names a driver. For a while, one machine can have both formats side by side.

What to do

Check which driver each container actually has, not only the default:

# The daemon default
docker info --format '{{.LoggingDriver}}'

# The driver of every container, running or not
docker ps -aq | xargs docker inspect --format '{{.Name}} {{.HostConfig.LogConfig.Type}}'

On servers running Docker Engine, the risk is the old one: json-file with no limit. Pick a default in /etc/docker/daemon.json. Use local if your collectors read through the Docker API:

{
  "log-driver": "local"
}

Or keep json-file for a file-tailing collector and give it a limit:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart the daemon, then recreate the containers so that they pick up the setting. As the measurements show, a size cap on json-file means the oldest lines are gone from disk. That is fine if a collector has already shipped them, and a loss if it has not.

On Docker Desktop, you probably need to do nothing. docker logs works with both drivers, and tools that use the same API should too. If you run a file-tailing collector on your laptop, set "log-driver": "json-file" in Docker Desktop's daemon settings, or move the collector to a source that reads through the Docker API. We could not see where 4.94 sets its new default, so run the docker info check above after the upgrade.

In Compose files, set the driver for each service if a service depends on it, so the laptop and the server do the same thing:

services:
  api:
    image: example/api:1.4
    logging:
      driver: json-file
      options:
        max-size: '10m'
        max-file: '3'

To clear logs that already fill a disk, see How to Clear Docker Container Logs Properly.

The-DevOps-Daily/docker-log-driver-check on GitHub

Summary

Docker Desktop 4.94 moved new Linux containers to the local log driver. That is the driver Docker's own docs recommend, and in our test it kept all 500,000 lines in 24 MB of files where unbounded json-file used 84 MB. Docker Engine on servers still defaults to json-file with no rotation, so set a default on purpose. If something reads the files under /var/lib/docker/containers, it needs json-file. If it reads through the Docker API, local is the better default for both laptops and servers. In both cases, check docker inspect, because a new default does nothing for containers that already exist.

So all three brokers can be both. We wanted to know what that means in practice, so we ran the same four failure cases against six setups on Kafka 4.3.1, RabbitMQ 4.3.6 and NATS 2.15.0:

• replay after a bad deploy
• one poison message
• more consumers than partitions
• the order of events after a retry

Two settings explained most of the results, whatever the broker: does the storage keep a message after it is handled, and does the consumer acknowledge each message or keep an offset? How many messages a consumer may hold at once explained the rest.

Two results surprised us:

• RabbitMQ: since RabbitMQ 4.3, a quorum queue's delivery limit ignores basic.nack(requeue=True). With x-delivery-limit set to 5 and a dead letter queue, a poison message was retried more than 6,000 times in 45 seconds. The same code on RabbitMQ 4.2.9 dead-lettered it after 6 deliveries. If your consumers requeue failed messages with a nack and rely on the delivery limit, upgrading to 4.3 turns off your poison message handling.
• Kafka: a share group with six consumers gave work to only three of them, because each consumer takes whole producer batches.

TLDR

• Choose two things, not a broker. Storage: does it keep messages after they are handled? Consumption: per-message acknowledgements, or an offset?
• Replay worked on every setup whose storage keeps messages, including setups with per-message acks: a Kafka share group and a NATS limits stream. It returned nothing on a RabbitMQ quorum queue and a NATS work-queue stream.
• Poison messages:
  • The two offset readers stopped at the poison message: 10 of 100 messages handled.
  • The Kafka share group and both NATS setups gave up on it after 5 deliveries and handled the other 99.
  • The RabbitMQ 4.3 quorum queue gave up only when the consumer used basic.reject. With basic.nack it looped. On RabbitMQ 4.2.9, both calls dead-lettered the message.
• More consumers than partitions:
  • The consumer group used 3 of 6 consumers, one per partition.
  • The share group also used only 3, until the producer sent one record per batch. Then all 6 got work.
  • Each consumer of a RabbitMQ stream read all 120 messages.
• Order after a retry:
  • The offset readers kept per-key order in 5 of 5 runs.
  • The per-message-ack setups lost it in every run where a consumer held several messages at once (prefetch 10, a fetch of 10, a share group batch), and kept it with one message in flight.
  • The Kafka share group lost it even with max.poll.records=1.

Prerequisites

• Docker, and Python 3.10 or later
• Basic familiarity with consumer groups and acknowledgements
• Optional: the demo repository, which runs every experiment in this post:

The-DevOps-Daily/queue-or-log on GitHub

Two choices, not one

A "queue" and a "log" each bundle two separate decisions. Pull them apart and the six setups we tested fit in one table:

Storage decides what happens after a message is handled. A quorum queue and a work-queue stream delete it. A Kafka topic, a RabbitMQ stream and a NATS limits stream keep it until retention removes it.

Consumption decides what a consumer remembers. An offset reader remembers a single position: everything before it is done. A per-message reader remembers each message's state, so one message can fail while the next one succeeds.

The fourth combination, storage that deletes handled messages read with an offset, does not exist. Kafka offers two of the three useful combinations, and RabbitMQ offers two through its two queue types. NATS also has ordered consumers, which read without acknowledgements and keep their own position, but we did not test them.

On a failure, each setup used its standard failure path:

• The share group released the record.
• The quorum queue got a basic.nack, and in a second run a basic.reject.
• NATS got a nak.
• The two offset readers went back to the failed offset and tried again. An offset reader has no broker-side way to set one message aside: if it commits past a message, the message stays on disk, but the reader counts it as done. An application can do more, for example keep reading ahead and track failed offsets itself, but then it builds its own retry bookkeeping. We kept to the plain retry.

The brokers ran as single nodes on a Raspberry Pi 4. The results describe behaviour, not throughput, and every number comes from the JSON files in the repository's runs/ directory.

the brokers under test

$ docker exec qol-kafka /opt/kafka/bin/kafka-topics.sh --version
4.3.1
$ docker exec qol-kafka /opt/kafka/bin/kafka-features.sh --bootstrap-server localhost:9092 describe | grep share
Feature: share.version                             SupportedMinVersion: 0                SupportedMaxVersion: 1                FinalizedVersionLevel: 1                Epoch: 7146
$ docker exec qol-rabbit rabbitmqctl version
4.3.6
$ docker exec qol-nats nats-server --version
nats-server: v2.15.0

Share groups need no extra setup on Kafka 4.3: share.version is finalized at level 1 on a new cluster.

Experiment 1: replay after a bad deploy

The scenario is common. A consumer shipped with a bug and handled 1,000 messages wrong, so after the fix you want those 1,000 messages again. Each setup handled 1,000 messages and acknowledged them all. Then we tried to read them a second time, in whatever way the setup allows.

Replay followed storage, not consumption. The share group and the NATS limits stream hand out work one message at a time, like a queue, and both replayed all 1,000, because the messages were still on disk. The quorum queue and the work-queue stream deleted each message when it was acknowledged, so after the first pass there was nothing to replay.

That changes the old advice. "Use Kafka if you might need to replay" was really "use storage that keeps messages". With share groups, you can have queue-style workers and keep the replay. The cost is the retention you pay for. The reset also has a rule: kafka-share-groups.sh --reset-offsets refuses to run while the group has active members, so you stop the workers first.

Experiment 2: one poison message

One message out of 100, number 10, made the handler fail every time. Each setup had 45 seconds and used its own failure path.

The two offset readers handled messages 0 to 9 and then stopped. Nothing behind the poison message moved for the rest of the 45 seconds, while the reader retried it 89 times (Kafka) and 928 times (RabbitMQ stream). This is not a broker bug. Without extra bookkeeping in your code, an offset reader has two options: retry in place and block the partition, or commit past the message, which leaves it on disk but out of sight unless your code copies it somewhere first. Kafka's dead letter topics today, such as the ones in Kafka Connect and Spring Kafka, are client or framework code making that second choice.

The share group did what a queue should do. It delivered the poison message 5 times, which is the default share.delivery.count.limit, then archived it and handled the other 99. Archived means no member of that share group gets it again. The record is not deleted: after the run, a plain consumer read offset 10 and got the poison message back. Kafka 4.3 does not copy it anywhere for you, though. KIP-1191, which the Kafka project has accepted, adds a dead-letter topic for share groups. It is not in 4.3.

NATS behaved the same way. After 5 deliveries, the consumer stopped delivering the message and the server published one $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES advisory, which we caught by subscribing during the run. The message stayed in the stream in both cases: in the work-queue stream it was the only one left, and in the limits stream all 100 remained.

RabbitMQ 4.3: nack no longer counts

The quorum queue rows are the ones to check in your own code. Quorum queues have had a delivery limit of 20 by default since RabbitMQ 4.0, and we set x-delivery-limit to 5 with a dead letter queue. With a consumer that failed with basic.nack(requeue=True), the poison message never reached the dead letter queue. It was retried 6,064 times in 45 seconds, about 135 times a second, and was still in the queue when the test ended. The same queue with basic.reject(requeue=True) dead-lettered it after 6 deliveries, which is the first delivery plus 5 redeliveries:

2_poison.py (excerpt of the recorded output)

$ ../.venv/bin/python 2_poison.py
    "rabbitmq quorum queue (limit 5 + dead letter, basic.nack)": {
      "others_handled": 99,
      "poison_attempts": 6064,
      "seconds": 45.0,
      "dead_letter_count": 0,
      "dead_letter_body": null,
      "messages_left": 1,
      "poison_ends": "still in the queue, retried in a loop"
    },
    "rabbitmq quorum queue (limit 5 + dead letter, basic.reject)": {
      "others_handled": 99,
      "poison_attempts": 6,
      "seconds": 4.1,
      "dead_letter_count": 1,
      "dead_letter_body": "10",
      "messages_left": 0,
      "poison_ends": "in the dead letter queue"
    },

This is new, and it is documented. The RabbitMQ 4.3 release post from April 23 says: "Up to RabbitMQ 4.2, every requeued message incremented its delivery-count by 1, regardless of the reason." From 4.3, the queue keeps two counters. acquired-count goes up on every requeue. delivery-count, which the limit is checked against, goes up only on a failed delivery, and a basic.nack does not count as one. In the post's words: "messages returned without marking the attempt as a failure no longer count toward the limit."

We ran the same test, limit 5 and a dead letter queue, on both versions:

So a consumer that requeues failed messages with basic.nack(requeue=True) and relies on the delivery limit handled poison messages correctly on 4.2, and loops on them after an upgrade to 4.3. Nacking is a common choice, because only basic.nack can return several messages at once. A nack with requeue=False is different: it dead-letters the message straight away, on both versions. Check which call your consumer or framework makes:

# Counts towards x-delivery-limit, so the message is dead-lettered
# (or dropped, without a DLX) once the limit is reached.
channel.basic_reject(delivery_tag, requeue=True)

# From RabbitMQ 4.3 this does NOT count: the same poison message
# comes straight back, forever. Up to 4.2 it counted.
channel.basic_nack(delivery_tag, requeue=True)

Warning

In our run, the loop retried the poison message about 135 times a second, as fast as the consumer could fail it. The queue depth stayed at one message and the other 99 flowed normally, so neither metric showed a problem. On a real consumer, each failure may also log an error or call a downstream service. Watch the redelivery rate as well as the queue depth.

Experiment 3: more consumers than partitions

120 messages, each taking 50 ms to handle, spread over a Kafka topic with 3 partitions. A single consumer needs about 6 seconds and six consumers about 1. We ran six consumers on each setup.

Consumers that got work, out of six

120 messages of 50 ms each. The Kafka topics had 3 partitions. Each RabbitMQ stream consumer read all 120 messages, so its six busy consumers did the same work six times. Recorded in runs/3_parallel.json.

The consumer group result is the one everybody knows: one partition goes to one member, so three of the six members sat idle.

The share group result is the surprise. Share groups exist so that more consumers than partitions can share the work, and here they did not. Three members got all 40 records of a partition each, and three got nothing. Setting max.poll.records to 1 changed nothing.

The cause is how a share group hands out records. In the default acquire mode, batch_optimized, a member acquires whole producer batches, and max.poll.records is only a soft cap. Our producer batched records the way producers normally do, so each partition's 40 records sat in a single batch, and whichever member fetched first took the lot. When we produced every record as its own batch, and kept max.poll.records at 1, the six members split the work almost evenly: 19 to 21 each, done in 1.3 seconds. We did not record the producer batch boundaries directly; the even split once every record was its own batch is the evidence.

Jack Vanlightly's series on share group parallelism explains this in depth, along with the other acquire mode, record_limit (KIP-1206). That mode makes max.poll.records a strict cap. We could not test it: librdkafka 2.15, which the Python client uses, refuses share.acquire.mode with "No such configuration property". His follow-up post also warns about fetch waits with record_limit. The practical lesson: with small backlogs and normal producer batching, a share group spreads work much less evenly than its name suggests.

The RabbitMQ stream result is a different thing again. Every consumer of a stream reads every message, the way each Kafka consumer group gets its own copy of a topic. Six consumers meant 720 handles of 120 messages. To split a stream's work across consumers, RabbitMQ has super streams, which partition a stream and work with single active consumer, and we did not test those.

The quorum queue, with prefetch 1, and both NATS setups, fetching one message at a time, spread the work evenly: 20 messages per consumer, done in about 1.1 seconds.

Experiment 4: order after a retry

Ten events for one account, e1 to e10, which must apply in order, like balance updates. The handler fails the first time it sees e3 and succeeds after that. A single consumer reads, so any reordering comes from the retry path alone. Retry timing varies, so every setup ran five times.

The offset readers kept order every time, because our readers retried the failed event before reading the next one. That is the simple way to use an offset, though not the only one.

When a consumer held several messages at once, the per-message-ack setups lost order in every run. The failed event went back to the broker while the consumer already held the next seven, so e3 was applied after e10. With exactly one message in flight, prefetch 1 on RabbitMQ or max_ack_pending 1 on NATS, they kept order in all five runs. NATS with one message per fetch but many allowed in flight was a race: e3 came back first in 2 runs, and e4 got in ahead of it in 3.

That result also corrects the claim we wrote before running the tests, which said per-message acknowledgements break order after a retry. They do not have to: limit the consumer to one message in flight and order survives. Several in flight permits reordering, and in our runs it happened every time the consumer held more than one.

The share group lost order even with max.poll.records=1, for the same reason as in experiment 3: the member acquired all ten records as one producer batch. With e3 released back to the group, the member had already accepted e4 to e10.

"One message in flight" has a price. It is exactly what an offset reader does on one partition. You get order, and you lose the parallelism that made you choose a queue. For per-key order with parallel workers, you need what a log gives you: route each key to a single place, one partition or one queue per key range, and keep one message in flight per key.

Picking a setup

The experiments reduce to a few questions. Work through them in order:

1. Will anyone need this message again after it is handled? That includes a replay after a bug, a new consumer next month, or rebuilding a read model. If yes, use storage that keeps messages: a Kafka topic, a RabbitMQ stream or a NATS limits stream. This is a storage choice, so you can still hand out work one message at a time, with a share group or a NATS consumer.
2. Does order per key matter? If yes, read with offsets, partitioned by key, or limit each key to one message in flight. Accept that a poison message then blocks its partition until code skips it. Decide in advance where skipped messages go.
3. Is each message an independent task? If yes, use per-message acknowledgements. You get retries, a delivery limit, a place for poison messages and parallelism beyond the partition count. Then check the two defaults that caught us:
   • RabbitMQ 4.3 and later: fail with basic.reject, not basic.nack, or the delivery limit never fires.
   • Kafka share groups: expect producer batches, not records, to be the unit of work, unless your client supports record_limit.

For the broker-by-broker settings:

Poison message settings, based on our runs

RabbitMQ

# A dead letter queue, then a quorum queue with a delivery limit
channel.queue_declare('orders.dead', durable=True, arguments={'x-queue-type': 'quorum'})
channel.queue_declare('orders', durable=True, arguments={
    'x-queue-type': 'quorum',
    'x-delivery-limit': 5,
    'x-dead-letter-exchange': '',
    'x-dead-letter-routing-key': 'orders.dead',
})

# On failure: reject, not nack, so the delivery counts on RabbitMQ 4.3+
channel.basic_reject(method.delivery_tag, requeue=True)

Kafka share group

# Group setting, once: delivery attempts before a record is archived (default 5)
#   kafka-configs.sh --bootstrap-server localhost:9092 --alter \
#     --entity-type groups --entity-name orders-workers \
#     --add-config share.delivery.count.limit=5

from confluent_kafka import AcknowledgeType, ShareConsumer

consumer = ShareConsumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'orders-workers',
    'share.acknowledgement.mode': 'explicit',
})
consumer.subscribe(['orders'])
while True:
    for msg in consumer.poll(1.0):
        if msg.error():
            continue
        try:
            handle(msg)
            consumer.acknowledge(msg, AcknowledgeType.ACCEPT)
        except Exception:
            # RELEASE retries it; REJECT would give up on it now
            consumer.acknowledge(msg, AcknowledgeType.RELEASE)
    consumer.commit_sync()

NATS JetStream

# Pull consumer that gives up after 5 deliveries
await js.add_consumer('ORDERS', ConsumerConfig(
    durable_name='workers',
    ack_policy=AckPolicy.EXPLICIT,
    max_deliver=5,
    ack_wait=30,  # seconds; our test runs used 2
))

# Then subscribe to the advisory to collect what was given up on:
# $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.workers

What we could not conclude

• Throughput and latency at scale. These are single-node brokers on a Pi. The results show what each setup does on a failure, not how fast it does it.
• Share groups in record_limit mode. The Python client cannot set share.acquire.mode. The Java client can, and that mode should spread small backlogs more evenly. We did not measure it.
• RabbitMQ super streams, and Kafka's coming share group dead-letter topics. Both would change parts of the tables above, and neither was in our setup.
• Clients other than the three we used: confluent-kafka 2.15.1, whose share consumer is still marked Preview, pika 1.3.2 and nats-py 2.16.0. A client that calls basic.reject for you will not hit the RabbitMQ loop.

Summary

The broker you run no longer decides whether you have a queue or a log. Each of these three does both. What decides the behaviour is a pair of settings that teams rarely write down: whether storage keeps a handled message, and whether consumers acknowledge messages one by one or keep an offset. Replay follows the first. Poison handling, parallelism and ordering follow the second, together with how many messages a consumer may hold at once.

Before you pick, write down which of the four failures you can live with. Then test the two settings that surprised us: since RabbitMQ 4.3, basic.nack does not count towards the delivery limit, and a Kafka share group hands out batches, not records. The scripts are in the repository, and each experiment runs in a few minutes against your own versions. For what happens when a worker dies halfway through a job, see running a background job that must not be lost. For when Kafka itself is the wrong tool, see 6 Apache Kafka use cases.

Upgrade notes tell you what changed. They do not show you your next plan, or the step at which a removed feature fails. So we tested it. We applied configurations with OpenTofu 1.12.7, ran 1.13.1 against the same state, and recorded what each command printed. This post goes through the results and ends with a checklist you can run before you change the version in CI.

TLDR

• base64gzip in 1.13.1 returns a different string for the same input. OpenTofu moved to Go 1.27, and Go changed its DEFLATE encoder. Both outputs decompress to identical bytes, but providers compare the string. With no config change, our plan was 1 to add, 1 to change, 1 to destroy.
• On aws_instance, a changed user_data_base64 means a stop/start, or a replacement if you set user_data_replace_on_change. On azurerm_linux_virtual_machine, a changed custom_data forces a new VM.
• ignore_changes hides the diff. In our test it also hid a real edit to the cloud-init file. Gzip done inside the cloudinit provider did not change at all.
• A winrm provisioner passes tofu validate and tofu plan on 1.13.1, with the old "will be removed in a future version" warning. It fails at apply, after the resource exists, and leaves the resource tainted.
• assumenotnull fixed a classic Invalid count argument error. assumestringprefix moved a mis-wired module input from a mid-apply failure to a plan error. Older versions reject these functions, and since 1.12 a required_version in a .tf file does not stop them.
• -lint=all is a useful experiment, but it only adds warnings. The exit code stays 0.

Prerequisites

• An OpenTofu 1.12.x codebase, or a Terraform codebase that you plan to move to OpenTofu
• A CI job or shell that can run tofu plan with read access to your real state
• jq, for the plan JSON recipe
• The 1.13.1 zip for your platform from the GitHub release page, checked against its SHA256SUMS file

How we tested

We ran everything on a Raspberry Pi with a 64-bit (arm64) OS: OpenTofu 1.12.7 and 1.13.1 for linux_arm64, the 1.13.1 linux_arm (32-bit) build, and Terraform 1.16.5 for comparison. Every zip matched its published SHA256 sum. In the outputs below, tofu-1.12.7 and tofu-1.13.1 are the two release binaries side by side.

We did not use a cloud account. The built-in terraform_data resource stands in for provider attributes. Its input argument updates in place when the value changes, and its triggers_replace argument forces a replacement. Those are the two ways real providers treat user data, which we checked in the provider docs. Every output in this post comes from these runs. Where we cut lines from an output, the block shows ... or the grep/tail we used, and long base64 strings are shortened with ....

base64gzip: same input, different string

base64gzip compresses a string with gzip and base64-encodes the result. Its most common job is cloud-init user data, because EC2 limits user data to 16 KB and compression gives you more room. Here is the same call on both versions, plus Terraform 1.16.5, and then a check of what a real payload (a 1,208-byte cloud-init file that installs nginx and writes a systemd unit) decompresses to:

base64gzip across versions

# same input, three binaries
$ echo 'base64gzip("hello")' | ./tofu-1.12.7 console
"H4sIAAAAAAAA/8pIzcnJBwAAAP//AQAA//+GphA2BQAAAA=="
$ echo 'base64gzip("hello")' | ./tofu-1.13.1 console
"H4sIAAAAAAAA/wAFAPr/aGVsbG8AAAD//wMAhqYQNgUAAAA="
$ echo 'base64gzip("hello")' | ./terraform-1.16.5 console
"H4sIAAAAAAAA/8pIzcnJBwAAAP//AQAA//+GphA2BQAAAA=="
# decompress the cloud-init payload from each version
$ echo 'base64gzip(file("cloud-init.yaml"))' | ./tofu-1.12.7 console | tr -d '"' | base64 -d | gunzip | sha256sum
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1  -
$ echo 'base64gzip(file("cloud-init.yaml"))' | ./tofu-1.13.1 console | tr -d '"' | base64 -d | gunzip | sha256sum
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1  -
$ sha256sum < cloud-init.yaml
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1  -

The encoded strings differ. The content does not: both outputs decompress to the exact bytes of the source file.

The cause is upstream. OpenTofu 1.12.7 is built with Go 1.26.6 and 1.13.1 with Go 1.27.1 (each binary records its Go version). The Go 1.27 release notes say that "the exact encoded output from Writer may be different from Go 1.26 as a result of the encoder implementation change", and that this carries through to compress/gzip. The OpenTofu 1.13 changelog calls the new output "equivalent to but not equal to" the output of earlier releases. The new output is stable: three runs of 1.13.1 gave the same string.

Terraform 1.16.5 is built with Go 1.26.8 and produced the same bytes as OpenTofu 1.12.7. So you also get this diff when you move from Terraform 1.16 to OpenTofu 1.13, not only when you upgrade OpenTofu.

What the plan shows

1. cloud-init.yaml unchanged
2. base64gzip() Go 1.27 DEFLATE
3. New string same bytes after gunzip
4. Provider diff string != state
5. Plan update or replace

This is the config we applied with 1.12.7:

locals {
  user_data = base64gzip(file("${path.module}/cloud-init.yaml"))
}

# Stands in for an attribute that a provider updates in place,
# such as user_data_base64 on aws_instance.
resource "terraform_data" "web_in_place" {
  input = local.user_data
}

# Stands in for an attribute that forces replacement,
# such as custom_data on azurerm_linux_virtual_machine.
resource "terraform_data" "web_replace" {
  triggers_replace = local.user_data
}

After tofu-1.12.7 apply, a 1.12.7 plan reported no changes. Then we planned with 1.13.1 against the same state and the same config:

$ tofu-1.13.1 plan
terraform_data.web_replace: Refreshing state... [id=4437d213-ef2e-bd8b-9c81-b5831d1b46c8]
terraform_data.web_in_place: Refreshing state... [id=b0ec5936-31c0-4653-0fb6-e353122675fa]

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place (current -> planned)
-/+ destroy and then create replacement

OpenTofu will perform the following actions:

  # terraform_data.web_in_place will be updated in-place
  ~ resource "terraform_data" "web_in_place" {
        id     = "b0ec5936-31c0-4653-0fb6-e353122675fa"
      ~ input  = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> "H4sIAAAAAAAA/5RTXW/jNhB89..."
      ~ output = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> (known after apply)
    }

  # terraform_data.web_replace must be replaced
-/+ resource "terraform_data" "web_replace" {
      ~ id               = "4437d213-ef2e-bd8b-9c81-b5831d1b46c8" -> (known after apply)
      ~ triggers_replace = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> "H4sIAAAAAAAA/5RTXW/jNhB89..."
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Two resources, no edits, one update and one replacement. On real resources, the result depends on the provider:

• aws_instance: for user_data_base64, where gzip output belongs, "Updates to this field will trigger a stop/start of the EC2 instance by default. If the user_data_replace_on_change is set then updates to this field will trigger a destroy and recreate of the EC2 instance."
• azurerm_linux_virtual_machine: for custom_data, "Changing this forces a new resource to be created."
• aws_launch_template: a changed user_data creates a new template version. Instances get it the next time your Auto Scaling group launches or refreshes from that version.

None of this is a disaster if you plan for it. But a stop/start of a single production box at 2 p.m. is the kind of surprise you want to catch in plan review, not after someone approves an apply because "it is only a version bump".

Find it before the upgrade

Start with a search. Run it after tofu init, so that it also covers registry and git modules in .terraform/modules:

grep -rnE --include='*.tf' --include='*.tofu' 'base64gzip\(|"winrm"' .

On our test folder it found both problems this post covers:

main.tf:2:  user_data = base64gzip(file("${path.module}/cloud-init.yaml"))
winrm/main.tf:6:      type     = "winrm"

A search only finds literal calls. A plan is the reliable check. Run a 1.13.1 plan in a throwaway job, save it, and list each changed attribute with this jq filter (save it as changed-attrs.jq):

.resource_changes[]
| select(.change.actions != ["no-op"])
| . as $r
| [ ($r.change.before // {}) | keys[]
    | select($r.change.before[.] != $r.change.after[.]
             and ($r.change.after_unknown[.] | not)) ]
| "\($r.change.actions | join("+"))  \($r.address)  changed: \(join(", "))"

$ tofu-1.13.1 plan -lock=false -out=upgrade.tfplan > /dev/null
$ tofu-1.13.1 show -json upgrade.tfplan | jq -r -f changed-attrs.jq
update  terraform_data.web_in_place  changed: input
delete+create  terraform_data.web_replace  changed: triggers_replace

A plan does not write state, and -lock=false stops a dry-run job from blocking a real apply. Remove the flag if you prefer to wait for the lock. You want a list in which every changed attribute is user data. Anything else in the list is real drift or a different upgrade change, so examine it separately.

Three ways to handle it

Handling the base64gzip diff

Accept it

# Plan with 1.13.1, review, and apply in a window you choose.
# The content is identical, so the only effect is the
# stop/start or replacement itself.
tofu plan -out=upgrade.tfplan
tofu show -json upgrade.tfplan | jq -r -f changed-attrs.jq
tofu apply upgrade.tfplan

Hide it (temporary)

resource "aws_instance" "web" {
  # ...
  user_data_base64 = base64gzip(file("${path.module}/cloud-init.yaml"))

  lifecycle {
    # TEMPORARY: OpenTofu 1.13 re-encodes base64gzip output.
    # This also hides real cloud-init edits. Remove it on the next change.
    ignore_changes = [user_data_base64]
  }
}

Gzip in the provider

data "cloudinit_config" "web" {
  gzip          = true
  base64_encode = true

  part {
    content_type = "text/cloud-config"
    content      = file("${path.module}/cloud-init.yaml")
  }
}

resource "aws_instance" "web" {
  # ...
  user_data_base64 = data.cloudinit_config.web.rendered
}

Accept it. This is usually the correct choice. Do the upgrade apply in a maintenance window, one environment at a time, and use the jq list as the change record.

Hide it. The release notes suggest ignore_changes as a temporary fix, and it works: with ignore_changes on both test resources, the 1.13.1 plan said No changes. Then we added a line to cloud-init.yaml and planned again. It still said No changes.

Warning

ignore_changes cannot tell the encoder change from a real edit. While it is in place, changes to your cloud-init file do not reach your instances, and the plan does not tell you. If you use it, open a ticket to remove it, and remove it on the next intentional user data change.

Move the gzip into the provider. With cloudinit_config and gzip = true, the compression runs in the provider binary, not in OpenTofu. We rendered the same file with hashicorp/cloudinit v2.4.1 under both 1.12.7 and 1.13.1, and the SHA256 of rendered was identical. This does not make you immune. It moves the dependency: v2.4.1 is built with Go 1.26.8, and a future provider release built with Go 1.27 can cause the same one-time diff. So pin the provider version and read its changelog. The switch itself also changes your user data once, because cloudinit_config wraps parts in a MIME multi-part document. Do it in the same window as the upgrade.

WinRM: passes validate and plan, fails at apply

OpenTofu 1.12 deprecated the winrm connection type, and 1.13 removed it (#4012) because "some of the upstream libraries OpenTofu was using to implement these features are no longer maintained". We expected tofu validate to report it. It does not. This is the test config (the host is a closed local port with a short timeout, so that the run fails fast):

resource "terraform_data" "bootstrap" {
  provisioner "remote-exec" {
    inline = ["powershell -Command Install-WindowsFeature Web-Server"]

    connection {
      type     = "winrm"
      host     = "127.0.0.1"
      user     = "Administrator"
      password = "example-only"
      https    = true
      timeout  = "10s"
    }
  }
}

$ tofu-1.13.1 validate
Warning: WinRM connection type is deprecated

  on main.tf line 6, in resource "terraform_data" "bootstrap":
   5:     connection {
   6:       type     = "winrm"

The winrm connection type is deprecated and will be removed in a future
version of OpenTofu.
...
Success! The configuration is valid, but there were some validation warnings
as shown above.

$ tofu-1.13.1 plan | grep 'Plan:'
Plan: 1 to add, 0 to change, 0 to destroy.

$ time tofu-1.13.1 apply -auto-approve
...
Error: remote-exec provisioner error

  with terraform_data.bootstrap,
  on main.tf line 2, in resource "terraform_data" "bootstrap":
   2:   provisioner "remote-exec" {

'winrm' connections are not supported in OpenTofu v1.13 or later

Error: Provisioners no longer support WinRM
...
real    0m0.153s

$ tofu-1.13.1 show | head -2
# terraform_data.bootstrap: (tainted)
resource "terraform_data" "bootstrap" {

1.13.1 still prints the 1.12 deprecation warning at validate and plan. It says the feature "will be removed in a future version", but the feature is already gone in this version. The error comes in under a second at apply. For comparison, 1.12.7 tried to connect to port 5986 and failed at the 10-second timeout, as it should.

The order matters. Provisioners run after the resource is created, and the OpenTofu docs say "if a creation-time provisioner fails, the resource is marked as tainted" and "will be planned for destruction and recreation upon the next tofu apply". With a real Windows VM, the VM is created and billed, then marked for replacement, and each apply after that recreates it and fails again until you remove the provisioner. Existing VMs whose provisioners ran long ago are not affected until something replaces them. Be careful with that last point: if a Windows VM uses gzipped custom_data, the base64gzip change above is exactly the kind of thing that replaces it, and then its WinRM provisioner runs again and fails.

To fix it, move to SSH or remove the provisioner:

• Windows Server 2019 and later can run OpenSSH Server. Set type = "ssh" and target_platform = "windows" in the connection block. If the SSH default shell is PowerShell, the connection docs also tell you to set script_path to a .ps1 path.
• Better, if you can: bake the configuration into the image, or run it from custom_data or user_data, so that no provisioner has to connect at all.

The grep above finds literal "winrm" strings. It does not find type = var.connection_type, so also search for connection blocks whose type comes from a variable.

32-bit builds: last series

The 1.13 changelog says this is "the final release series that will include official builds for 32-bit CPU architectures" (*_386 and *_arm). The Pi's 64-bit kernel can run 32-bit ARM binaries, so we ran the linux_arm build:

$ uname -m
aarch64
$ tofu-1.13.1-linux-arm version
OpenTofu v1.13.1
on linux_arm
$ tofu-1.13.1-linux-arm init

Warning: Support for 32-bit CPU architectures is ending soon

OpenTofu v1.13 is the last release series that will include official release
packages for 32-bit CPU architectures.

We recommend planning to migrate to a 64-bit CPU architecture instead.
Alternatively, you could build OpenTofu for linux_arm from source code
yourself, ...

Look at the second line of tofu version on each runner. If it says on linux_arm or on linux_386, that runner has to move. Common cases are 32-bit Raspberry Pi OS runners, old ARMv7 boards, and i386 container images. As the test shows, a 64-bit kernel does not help if the image or the binary you install is 32-bit. You have time: per the changelogs, the 1.13 series is supported until August 1, 2027, and 1.12 until February 1, 2027. The 1.11 series lost support on August 1, 2026.

The new functions: hints for unknown values

This is the main feature of the release. During a plan, any value that an API assigns at creation time is unknown, and OpenTofu cannot use an unknown value to decide how many instances to create. Here is the classic case: a network module creates a VPC, and a second module creates flow logs only when it gets a VPC ID.

# modules/network/main.tf
# terraform_data stands in for aws_vpc: its output is unknown until apply,
# the same way a VPC id is decided by the AWS API at creation time.
resource "terraform_data" "vpc" {
  input = "vpc-0a1b2c3d4e5f60718"
}

output "vpc_id" {
  value = terraform_data.vpc.output
}

# modules/flow_logs/main.tf
variable "vpc_id" {
  type    = string
  default = null
}

resource "terraform_data" "flow_log" {
  count = var.vpc_id != null ? 1 : 0
  input = var.vpc_id
}

On a first plan, 1.12.7 and 1.13.1 fail the same way:

Error: Invalid count argument

  on modules/flow_logs/main.tf line 7, in resource "terraform_data" "flow_log":
   7:   count = var.vpc_id != null ? 1 : 0

The "count" value depends on resource attributes that cannot be determined
until apply, so OpenTofu cannot predict how many instances will be created.
...

The assume... functions let the module author state what is true about the value even before it exists. Our first attempt failed:

Error: Invalid function argument

  on modules/network/main.tf line 8, in output "vpc_id":
   8:   value = assumenotnull(terraform_data.vpc.output)

Invalid value for "value" parameter: given value must have a known type;
consider using the \"convert\" function to specify a type to assume.

terraform_data.output takes on the type of input, so its type is also unknown during the plan. Most provider attributes, such as aws_vpc.id, are typed strings and do not have this problem. For a value like this, the docs tell you to combine the hint with the new convert function:

output "vpc_id" {
  value = assumenotnull(convert(terraform_data.vpc.output, string))
}

$ tofu-1.13.1 plan | grep -E 'will be created|Plan:'
  # module.flow_logs.terraform_data.flow_log[0] will be created
  # module.network.terraform_data.vpc will be created
Plan: 2 to add, 0 to change, 0 to destroy.

$ tofu-1.13.1 apply -auto-approve | tail -1
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The usual workaround for this error is a two-step apply with -exclude (the error message itself suggests it). Here, one plan is enough.

A hint is a promise, and OpenTofu checks it. The docs say that if the value turns out to be null, "the function raises an error", so the apply fails. Use a hint only where the provider really guarantees it, for example that an ID is never null after create.

Catch wiring mistakes at plan time

assumestringprefix is useful together with variable validation. Here the caller connects the subnet output to an input that expects a VPC ID, a mistake that is easy to miss in review:

# main.tf
module "flow_logs" {
  source = "./modules/flow_logs"
  vpc_id = module.network.subnet_id # wrong output wired in
}

# modules/flow_logs/main.tf
variable "vpc_id" {
  type = string

  validation {
    condition     = startswith(var.vpc_id, "vpc-")
    error_message = "vpc_id must be a VPC id (vpc-...)."
  }
}

Without hints, the plan passes because the value is unknown, and validation waits for apply. The apply created the VPC and the subnet, and then failed. Both outputs are filtered to the key lines, and the IDs are shortened:

$ tofu-1.13.1 plan
Plan: 3 to add, 0 to change, 0 to destroy.
$ tofu-1.13.1 apply -auto-approve
module.network.terraform_data.subnet: Creation complete after 0s [id=...]
module.network.terraform_data.vpc: Creation complete after 0s [id=...]
Error: Invalid value for variable
vpc_id must be a VPC id (vpc-...).
$ tofu-1.13.1 state list
module.network.terraform_data.subnet
module.network.terraform_data.vpc

Then we added prefix hints to the network module outputs:

output "vpc_id" {
  value = assumenotnull(assumestringprefix(convert(terraform_data.vpc.output, string), "vpc-"))
}

output "subnet_id" {
  value = assumenotnull(assumestringprefix(convert(terraform_data.subnet.output, string), "subnet-"))
}

Now the same mistake fails the plan, before anything is created:

$ tofu-1.13.1 plan
Error: Invalid value for variable

  on main.tf line 7, in module "flow_logs":
   7:   vpc_id = module.network.subnet_id # wrong output wired in
    ├────────────────
    │ var.vpc_id is a string

vpc_id must be a VPC id (vpc-...).

This was checked by the validation rule at modules/flow_logs/main.tf:4,3-13.

If you maintain shared modules, the outputs of your network, IAM and DNS modules are the best places for these hints. Callers get the benefit without changing their own code. assumeequal goes further: the docs show it with the AWS provider's arn_build function to make an IAM role ARN fully known at plan time, so that policy checks can see the real policy document.

Guard the module version correctly

A module that uses these functions does not work on older versions, and the errors are not clear. On 1.12.7, assumenotnull(...) alone gives Call to unknown function. With convert(..., string), the error is Invalid reference, because 1.12 reads string as a resource address. Terraform 1.16.5 also gives Call to unknown function.

You would usually add required_version = ">= 1.13.0" to a terraform block. That does not work here. Since 1.12, OpenTofu ignores required_version in .tf files and honors it only in .tofu files (see the RFC tracking issue and the settings docs). We tested this with a constraint that no version can meet:

So put the guard in a .tofu file:

# versions.tofu (the language block needs OpenTofu 1.12 or later)
language {
  compatible_with {
    opentofu = ">= 1.13"
  }
}

On 1.12.7 this gives This module is not compatible with OpenTofu v1.12.7, which is the clear message you want. If a module must also work with Terraform, use the .tofu precedence rule: when outputs.tf and outputs.tofu are both present, OpenTofu loads only the .tofu file. We put a hinted output in outputs.tofu and a plain one in outputs.tf. tofu validate (1.13.1) and terraform validate (1.16.5) both passed.

The lint experiment

validate, plan, apply and refresh accept a new -lint flag. 1.13 has four rules, all for the root module only: core:no-type-variable, core:unused-variable, core:unused-local, and core:count-instead-enabled. The last one suggests the enabled lifecycle argument instead of count = cond ? 1 : 0. We ran the experiment on a small file that breaks all four rules:

$ tofu-1.13.1 validate -lint=all -json | jq -r '.diagnostics[] | select(.severity == "warning") | .summary'
Experimental linting enabled
Input variable not used (core:unused-variable)
Local value not used (core:unused-local)
Could use enabled instead of count (core:count-instead-enabled)
Variable with no type (core:no-type-variable)

The exit code was 0. Lint results are warnings, and OpenTofu always adds an "Experimental linting enabled" warning. You can turn off one rule with !, for example -lint='all,!core:unused-variable'. The linting docs say that rules can change even in minor releases. For now, make it a non-blocking CI step and read the output. Do not gate merges on it yet.

Smaller changes worth a line

• Plan text. After No changes. Your infrastructure matches the configuration., 1.12.7 printed one more paragraph ("OpenTofu has compared your real infrastructure ... no changes are needed."). 1.13.1 does not. If a script greps for that paragraph, change it to use tofu plan -detailed-exitcode, which returns 0 for no changes, 1 for errors, and 2 for changes. Our upgrade plan returned 2.
• Platforms. Windows on ARM64 is now an official platform, and macOS builds require macOS 13 Ventura or later.
• State encryption. The aws_kms key provider accepts an encryption_context. gcp_kms accepts additional_authenticated_data, and openbao accepts associated_data.
• Crash recovery. A Go panic now writes a partial errored.tfstate to help you recover.
• Saved plans include the provider schemas, so tofu show on a plan file usually does not need to start providers.
• If you stay on 1.12 for now, take 1.12.7. It fixes a deadlock that an attacker-controlled SSH server could cause in remote-exec and file provisioners (CVE-2026-78662).

Upgrade checklist

1. After tofu init, search your code and .terraform/modules for base64gzip( and "winrm", and look for connection blocks whose type comes from a variable.
2. Remove WinRM provisioners before you upgrade. Validate and plan will not stop you, and a failure at apply taints the resource.
3. Run a 1.13.1 plan against real state in a throwaway job, save it, and run the jq filter. Every changed attribute should be user data.
4. For each user data change, decide: accept the stop/start or replacement in a window, or use a temporary ignore_changes with a ticket to remove it.
5. Check tofu version on every runner and workstation image, and move anything on linux_arm or linux_386 to 64-bit before August 1, 2027.
6. Change scripts that read plan text to use -detailed-exitcode.
7. In shared modules, add assume... hints to outputs whose IDs callers use in count, for_each or validation. Put a language block in a .tofu file to guard them.
8. Add -lint=all to CI as a non-blocking step and see what it finds.

The upgrade is not hard, but two of these changes are not visible until a specific step: the base64gzip diff shows only in a plan against real state, and the WinRM removal shows only at apply. If you run steps 1 to 3 first, you see both before they affect a real server.

Shopify's engineering blog describes the rehearsal in unusual detail: load tests at 150% of last year's peak, capacity planned with the cloud providers months ahead, a throttle at the edge that queues buyers when checkout is full, and inventory reserved during payment so two buyers cannot claim the last unit. This post walks through each from Shopify's own write-ups, then rebuilds the checkout half on a Raspberry Pi with k6 to measure what the last two steps buy.

The short version: checking stock before payment oversold 41 to 47 units of 500 in three runs, and reserving the unit first sold exactly 500. When buyers arrived twice as fast as payment could take them, a plain queue charged more shoppers after they had given up than it confirmed purchases for. A simple waiting room kept checkout under 400 ms at p95 instead, at the cost of turning the overflow into shoppers who gave up waiting.

TL;DR

• Shopify rehearses all year: its load generator, Genghis, runs checkout flows in production with flash-sale bursts on top, at 150% of last year's load, and capacity is planned months ahead.
• When checkout is full, an edge throttle queues buyers. The first version was a lottery that left some waiting 40 minutes; a signed first-attempt timestamp made it fair.
• Inventory is reserved when payment starts and claimed when it succeeds.
• In our runs the oversell from check-then-pay was roughly arrival rate times payment time: lower traffic shrank it but did not remove the race. At twice the payment capacity, the question was not how many orders got through but who got them.
• Our first batch of runs quietly skipped up to a fifth of its shoppers. Fail arrival-rate load tests on dropped_iterations.

Prerequisites

• Comfort with HTTP, SQL transactions and the idea of a load test
• To run the demo: Node.js 22.13 or newer (for the built-in node:sqlite) and the k6 binary; no Docker, no database server
• About 25 minutes of machine time for the full set of recorded runs

A flash sale is an overload you can schedule

Bart de Water, who worked on Shopify Payments, defined the term in his QCon talk on Shopify's flash-sale architecture: "A flash sale is a sale for a limited amount of time, often with limited stock. It's over in a flash because the product can sell out in seconds, even if there are thousands of items in inventory." He also draws the line that matters here: "Storefront is mostly about read traffic, while our checkout does most of the writing and has to interact with external systems as well."

Reads cache well. Writes that call a payment provider do not, and that is where Shopify got hurt. The first of two posts on its checkout throttle describes Kylie Cosmetics running sales that sold out quickly, roughly every week, and one in February 2016 that "took down not just her store, but all others on the database shard where her store was allocated." What followed was a set of habits Shopify still writes about, and de Water's talk has the one-line reason they keep paying off: "today's flash sale will be tomorrow's base load."

Those habits answer three questions. Will the platform hold at the peak? What happens to buyers who arrive when checkout is full? Does the one write that matters stay correct under contention?

Load testing as a discipline, not a launch task

Shopify's 2025 BFCM readiness post opens with "Bimonthly fire drills all year, simulating 150% of last year's BFCM load." The tool is in-house:

Our load testing tool Genghis runs scripted workflows that mimic user behavior like browsing, cart adds, and checkout flows. We gradually ramp traffic to find breaking points. Tests run on production infrastructure simultaneously from three GCP regions (us-central, us-east, and europe-west4) to simulate global traffic patterns. We inject flash sale bursts on top of baseline load to test peak capacity.

Five major scale tests ran from April to October 2025. The fourth reached 146 million requests per minute and more than 80,000 checkouts per minute; the last went to the p99 forecast of 200 million requests per minute. Early tests found that "core operations threw errors and checkout queues backed up", and adding authenticated checkout "exposed rate-limit paths that anonymous browsing never touches."

Three details worth copying at any scale:

• Production-shaped targets. De Water's talk describes Genghis hitting benchmark stores in production, at least one per pod, "at least weekly", paying through a benchmark gateway that "can respond with both successful and failed payments with a realistic distribution of response time latencies that we see in production." The demo copies that idea.
• The burst, not just the volume. A 2023 post lists a flash-sale flow in which "simulated users purchase a single product", plus an "abort switch" that stops every test at once.
• Written-down failure. Toxiproxy, Shopify's open source tool for simulating network conditions, injects failures during load tests, and findings go into a Resiliency Matrix of failure scenarios, recovery objectives and runbooks.

Pick the load model before the tool

What decides whether a load test can see a flash sale at all is the workload model. k6's page on open and closed models states the trap: "When the target system is stressed and starts to respond more slowly, a closed model load test will wait, resulting in increased iteration durations and a tapering off of the arrival rate of new VU iterations." A closed test slows down exactly when the system does and reports a calm result. Buyers at a drop do not wait for the previous buyer's page to load, so you need an open model, where arrivals follow a schedule whatever the server is doing.

All three common open source tools can drive an open model and fail a CI job; k6 and Gatling also have explicit closed-model options:

On merit: k6 is a single Go binary whose thresholds work on custom metrics, which made "fail if any unit was sold twice" a one-line rule in the demo. Gatling separates open and closed injection in its API, so the model is explicit in every script, and suits JVM teams. Artillery's YAML scenarios are the quickest to write for plain HTTP, and its Playwright engine drives real browsers when the checkout page's JavaScript matters. Any of them could run the demo.

Pre-scale for the peak you know is coming

Shopify's 2025 preparation started in March with capacity planning, including "submitting our estimates to our cloud providers so they don't run out of cloud." In the uncertain year of 2020, Capacity Planning at Scale records the choice: "We decided to scale to our more aggressive growth scenarios to ensure our platform is stable regardless of what happens." Before scale tests, components are brought up to a "BFCM profile" ahead of time.

Change is managed the same way. A 2018 post describes a feature freeze that "starts several weeks before BFCM" and a code freeze a few days before. The 2025 post puts it as a rule: "We don't use BFCM as a release deadline." Architectural changes and migrations land months earlier.

Why not let autoscaling handle it? Shopify's posts do not say they turn it off, and we found no primary source that does. But in the demo below the stock is gone about six seconds into the sale, while a Kubernetes Horizontal Pod Autoscaler checks metrics every 15 seconds by default, before a new pod is even scheduled. Autoscaling suits the slow tides of a long weekend; for the first minute of a drop, the capacity has to exist already. Our Kubernetes HPA lab and horizontal vs vertical scaling simulator let you feel that lag safely.

Some capacity cannot be pre-scaled at all because it belongs to someone else, like a payment provider. The second experiment is about what to do when arrivals exceed it.

Put the queue at the edge, not inside checkout

After the Kylie outage, Shopify had a week before the next sale. Part I explains why a plain rate limit would not do: "For customers anything that looked like the website crashing would be interpreted as such." The team built a throttle into its Nginx and OpenResty edge tier:

The solution we landed on was to throttle users using a leaky bucket algorithm built into our edge tier. [...] The number of requests served would be reset every period (in our case, 5 seconds), and it was up to us to inform the client when to retry a rejected request.

Buyers over the limit saw a queue page that polled /checkout; those who got through received a signed cookie to skip the throttle for the session. The platform held, and then buyers complained of waiting up to 40 minutes for a 40-minute sale. The post admits that "in reality they were randomly polling the throttle", a lottery.

Part II fixed fairness without adding state to the edge. A buyer's first checkout attempt is stamped with a timestamp in a signed cookie, and each load balancer computes a threshold, "the virtual version of the 'Now Serving: 42' counters at delis", that lets the earliest timestamps through to the leaky bucket. A feedback controller moves the threshold; after simulations, the team kept only its proportional term.

1. Buyer clicks checkout
2. Edge throttle Nginx + Lua
3. Checkout writes + payment

Outcomes:

• under capacity: signed cookie, straight through
• over capacity: queue page, poll, earliest timestamp first

Five years later, de Water's 10 tips for resilient payment systems (2022) still describes "scriptable load balancers to throttle the amount of checkouts happening at any given time", with a waiting queue when demand exceeds capacity. It explains why with Little's Law and a sentence worth pinning above any capacity plan: "your application can't out scale the world." At the edge, a waiting buyer costs a cached page and a poll, not a worker, a database connection and a payment slot.

Protect the one write that must not go wrong

The 2025 post calls checkout, payment processing, order creation and fulfillment "critical journeys". Older parts of the architecture exist to keep them alive:

• Pods. A pod "consists of a set of shops that live on a fully isolated set of datastores", which limits how far one shop's sale can reach. De Water adds that some extra-large merchants get a pod to themselves, and other systems stop a big merchant's flash sale from trying to "monopolize all the capacity" of a shared pod.
• Semian. Shopify's circuit breaker and bulkhead library for Ruby starts from the fact that slow resources fail slowly, and while threads wait on one, "the slow resource has caused a cascading failure by occupying workers and therefore losing capacity."
• Inventory reservations, the part the demo rebuilds.

Shopify's May 2026 post, We replaced Redis with MySQL for inventory reservations, says oversell protection works "by reserving inventory during payment processing", which it calls "a short hold that prevents two concurrent checkouts from claiming the same unit." Reserve holds items when payment starts; claim deducts them from the ledger when payment succeeds. The reservation used to be a Redis DECR, but then the claim step had to update the MySQL ledger and clean up Redis, two operations that could not be wrapped in one atomic step. In MySQL, the obvious schema failed: "A single row with a quantity column couldn't handle the contention." The shipped design uses one row per sellable unit, taken with SELECT ... FOR UPDATE SKIP LOCKED so concurrent checkouts take different rows, from a pool capped at 1,000 rows per item and location that a replenishment process refills. The post also ties this article's two halves together: "Slow reservations trigger throttling and a worse buyer experience." Remember the single-row detail; the demo's fix is the single-row version.

Rebuilding the checkout half on a Raspberry Pi

The demo is a small checkout service and one k6 scenario, built to answer two questions a skeptical reader can check: does reserving before payment matter at modest traffic, and what does a queue in front of checkout change when arrivals exceed what payment can process?

The-DevOps-Daily/flash-sale-playbook on GitHub

Everything ran on a 4-core Raspberry Pi 4 that was also doing other work, with k6 2.3.0 and the service on the same machine, so absolute numbers are small. Every variant ran the same scenario on the same machine, interleaved with the others, and each run records the load average before and after it.

• The service is one Node.js 24 process, one product, and an SQLite database in memory-backed storage. One process stands in for a fleet: the await between the read and the write lets other requests run in between, as requests on separate servers would.
• The payment provider is a stand-in that takes 100 to 400 ms (uniform, seeded per run) and declines 5%. In the second experiment it also has a fixed number of slots.
• The scenario is a k6 ramping-arrival-rate executor with every VU created before the sale. Each iteration is one shopper. After the sale, teardown() asks the server what it sold and reports it as metrics, so thresholds fail the run on correctness, not just latency.

// k6/flash-sale.js, trimmed
export const options = {
  scenarios: {
    flash_sale: {
      executor: 'ramping-arrival-rate',
      startRate: 0,
      timeUnit: '1s',
      preAllocatedVUs: profile.vus,
      maxVUs: profile.vus,
      stages: profile.stages, // drop: 0 -> 300/s over 10s, hold 20s, down over 5s
    },
  },
  thresholds: {
    dropped_iterations: ['count==0'], // k6 kept its own schedule
    oversold_units: ['count==0'], // reported by teardown() from the server
    orders_to_clients_that_left: ['count==0'],
    'http_req_duration{name:checkout}': ['p(95)<2000'],
  },
};

Before the recorded runs, scripts/predict.mjs simulated every variant 400 times with an idealized copy of k6's arrival schedule (it has one extra arrival at the very end) and the same payment model, ignoring server CPU time. Its output from the start of the first batch, kept with the discarded runs, is identical to the one used here. Every recorded order count below fell inside the simulated 5th to 95th percentile range; a few latency figures landed just outside it, which is unsurprising for a model with no CPU time.

Experiment 1: check, pay, then write

The naive checkout is the order most people write first (simplified from server.mjs):

async function checkoutNaive(shopper) {
  const { stock } = q.readStock.get(SKU); // SELECT stock ...
  if (stock <= 0) return { status: 409 }; // sold out
  const payment = await authorizePayment(); // 100-400 ms; other requests run here
  if (payment === 'declined') return { status: 402 };
  q.decrement.run(SKU); // UPDATE inventory SET stock = stock - 1
  recordOrder(shopper);
  return { status: 200 };
}

The fix moves a conditional decrement in front of payment, so the database decides who gets each unit:

async function checkoutReserve(shopper) {
  // UPDATE inventory SET stock = stock - 1 WHERE sku = ? AND stock > 0
  const { changes } = q.reserve.run(SKU);
  if (changes === 0) return { status: 409 }; // sold out, no payment call
  const payment = await authorizePayment();
  if (payment !== 'paid') {
    q.release.run(SKU); // give the unit back
    return { status: 402 };
  }
  recordOrder(shopper);
  return { status: 200 };
}

The drop profile ramps from zero to 300 new shoppers a second over 10 seconds, holds for 20 and ramps down over 5: 8,249 shoppers for 500 units. Each mode ran three times, interleaved to spread background load across both:

Every run started all its scheduled shoppers and used the same code, stamped by hash in its run.json. The p95 covers every checkout request, including the quick 409 sold-out answers most shoppers got.

Units sold beyond the 500 in stock

Recorded k6 runs on a Raspberry Pi 4. Tick marks show the mean of 400 simulated runs from scripts/predict.mjs. Reserve sold exactly 500 every time.

flash-sale-playbook

# two of the recorded runs, as scripts/run-all.sh started them
$ scripts/run.sh drop-naive-1 drop MODE=naive SEED=1
drop-naive-1: k6 exit 99, results in results/drop-naive-1
$ scripts/run.sh drop-reserve-1 drop MODE=reserve SEED=1
drop-reserve-1: k6 exit 0, results in results/drop-reserve-1
# what the server says it sold
$ jq -c '{mode: .config.mode, stockInitial, orders, oversold, stockNow}' results/drop-naive-1/server-stats.json
{"mode":"naive","stockInitial":500,"orders":544,"oversold":44,"stockNow":-44}
$ jq -c '{mode: .config.mode, stockInitial, orders, oversold, stockNow}' results/drop-reserve-1/server-stats.json
{"mode":"reserve","stockInitial":500,"orders":500,"oversold":0,"stockNow":0}
# why the naive run exited 99
$ grep -A14 THRESHOLDS results/drop-naive-1/k6-output.txt
  █ THRESHOLDS 

    dropped_iterations
    ✓ 'count==0' count=0

    http_req_duration{name:checkout}
    ✓ 'p(95)<2000' p(95)=189.78ms

    orders_to_clients_that_left
    ✓ 'count==0' count=0

    oversold_units
    ✗ 'count==0' count=44

Every naive drop run sold units that did not exist, 8 to 9% more than the stock, and left the counter negative, so nothing downstream would have stopped those orders. Every reservation run sold exactly 500. Latency could not tell them apart (most shoppers arrived after the sellout and got a quick 409), so a load test that only checked p95 and errors would have passed both.

The size of the oversell is predictable. Every shopper who passes the stock check before the counter reaches zero gets an order, and the counter only moves when a payment completes. So at the moment it hits zero, about (arrival rate × average payment time × share of payments approved) shoppers are still in flight. The 500th naive order landed about 6.2 seconds after the sale opened, with arrivals near 185 a second: 185 × 0.25 s × 0.95 ≈ 44. The simulation predicted a mean of 42.5, with 5th to 95th percentiles of 37 and 48; the runs landed at 44, 41 and 47.

Lower traffic shrinks it but does not remove the race: at a tenth of the arrival rate, the prediction was 4 to 9 and the run oversold 8 (1.6% of the stock). A slower payment step makes it larger. The bug is about how many payments are in flight when the last unit goes, not about Black Friday traffic.

Experiment 2: more buyers than payment can take

The second scenario is the BFCM shape: plenty of stock, arrivals beyond what checkout can process. Payment has 8 slots and averages 250 ms, so it completes about 32 checkouts a second (Little's Law: 8 / 0.25 s). The peak profile ramps to 64 shoppers a second, twice that, holds for 30 seconds and ramps down over 10: 2,559 shoppers. Each will spend at most 15 seconds trying to buy, in one slow request or several tries.

Three variants, one change each, on the reservation code:

• Queue: no protection. Requests wait for a payment slot as long as it takes.
• Deadline: the baseline a hostile reader would ask for. k6 sends the time the shopper gives up, and the server skips the charge (504) when less than the stand-in's maximum payment time, 400 ms, is left.
• Waiting room: at most 8 checkouts in progress; everyone else gets 503 with Retry-After: 1 and retries after a jittered half to one and a half seconds, giving up when the next wait would take them past 15 seconds. This is Shopify's random-polling first version, the lottery, not the timestamp fix.

The first five rows add up to all 2,559 shoppers in every run. The charges in the sixth row overlap with the timeouts: in the queue variant, most shoppers who saw a timeout were charged anyway.

2,559 shoppers, payment capacity about 32 checkouts a second

Mean of three recorded k6 runs per variant on a Raspberry Pi 4. Tick marks show the mean of 400 simulated runs from scripts/predict.mjs.

Through the 45 seconds of overload, every variant completed 29 to 31 orders a second, by the server's own timestamps. Payment was the limit and none of the three changed it. What changed was who got those orders, and how long everyone waited.

The queue turned the overload into charges nobody saw. The line for payment grew past 1,100 requests; once the wait passed 15 seconds, nearly every request was served after its shopper had given up. Of about 2,440 orders per run, roughly 1,390 went to shoppers whose browser had shown a timeout, more than the 1,050 or so who saw a confirmation. In a real shop, each is a card charged for a purchase the buyer thinks failed.

The deadline removed those charges and sold the most, about 1,810. The cost is time: the queue still settles near the patience limit, so the median buyer waited 12.5 to 13.1 seconds, with up to 949 requests open at once. In a thread-per-request server each would also hold a worker, the failure Semian exists to prevent.

The waiting room kept checkout fast: 8 in progress at most, under 400 ms at p95, and a median time to purchase of 4.3 to 5.0 seconds including the waiting. Between 826 and 847 shoppers per run gave up after close to 15 seconds of "please wait", and the server answered about 22,000 cheap 503s, around 8.6 per shopper. That is the trade Shopify made by moving the wait to a cached page at the edge.

It confirmed about 10% fewer purchases than the deadline queue, and the timestamps show where most of that gap built up. Both sold about 30 a second through the overload. After arrivals stopped, 50 seconds into the sale, the deadline queue kept serving its newest arrivals, who still had patience left, until about 62 seconds; the waiting room ran dry at about 58. Server-recorded orders after the 50-second mark account for about 154 of the roughly 175-order gap. The cause is this particular polling policy rather than waiting rooms in general: admission is random, so shoppers who had waited longest gave up first; a shopper gives up as soon as the next random wait would cross the 15-second limit, up to 1.5 seconds early; and a freed slot sits idle until someone's retry arrives. It is Shopify's Part I lesson in miniature: random retries spend the patience of the people who came first. The 10 to 14 late charges are shoppers admitted with less patience left than their payment took. Adding the deadline check to the waiting room should remove most of them; we did not run that combination.

The bug in our own load test

Our first full batch of runs, kept under results/discarded/, started with 50 pre-allocated VUs and let k6 create more on demand. In the peak runs every waiting shopper holds a VU; k6 could not create them fast enough and skipped the shoppers it could not start on time: 541, 542 and 440 of 2,559. The runs completed, the summaries looked plausible, and the queue variant looked far better than it should, because it had faced a lighter sale.

k6 reports this as dropped_iterations, and our summary script flagged it. The fix was to create every VU up front and fail the run on dropped_iterations: ['count==0']. Later, a burst of unrelated work on the Pi pushed the load average near 9 on four cores. One peak run in that window skipped 253 shoppers, and another started 2,557 of 2,559 without reporting any as dropped, which only surfaced when the summary script compared runs. Both were re-run once the machine was quiet and kept with a note. The rule for throwing a run away was incomplete delivery of the schedule, not load: peak-queue-3 started under a load average of 8.9, delivered all 2,559 shoppers, landed inside its predicted ranges and was kept. Before you believe anything an arrival-rate test says, check that it delivered its schedule.

What this does not show

• Scale. A few hundred requests a second on one small machine, with k6 and the server sharing four cores. The direction of each result should hold; the absolute numbers do not transfer.
• Hot-row contention. SQLite serializes writes, so the single-row conditional UPDATE never fought for a row lock the way it would in MySQL or Postgres. Shopify says that design "couldn't handle the contention" at its scale. The demo shows that reserving before payment is necessary, not that one row is enough.
• The cost of a held request. Node.js holds a waiting connection cheaply. In a thread-per-request server, the queue and deadline variants would look worse.
• A fair waiting room. Ours is the lottery Shopify replaced. A timestamp-ordered queue should change who gets served and narrow the spread of waits; we did not build one.
• Real payment latency. The stand-in and the predictor use a uniform 100 to 400 ms with no long tail. Real providers have tails, which make every queue worse.
• Rare cases. Three runs per variant (one for low) agree with each other, and their order counts sit inside the simulated ranges, but three runs cannot rule out an occasional bad one.

The playbook, condensed

1. Rehearse the shape, not just the volume. Model the drop as a burst on baseline traffic with an open-model load test, and run it on a schedule.
2. Make the load test check correctness. After the run, ask the system what it sold. Fail on oversells, on charges to buyers who had left and on dropped arrivals, not only on p95.
3. Have the capacity before the minute you need it. Forecast, agree capacity with providers, scale up ahead of the peak and stop risky changes well before it.
4. Bound the work that reaches checkout. Admit what payment and inventory can finish and park everyone else somewhere cheap, ideally at the edge and in arrival order.
5. Reserve before you charge, and release on failure. The database decides who gets the last unit, not the application's memory of an earlier read.

Shopify's own posts are worth reading in full, starting with the 2025 readiness program, the checkout throttle story and the inventory reservations post. For the neighbouring problems, our post on how Stripe avoids double-charging covers retries around the payment call, and the rate limit simulator shows how leaky and token buckets behave under bursts.

It is the second disclosure of this kind in five weeks. On September 1, Manifold Security published GitSpawn: eight findings across seven coding agents (Claude Code, Codex, Cursor, goose, Qwen Code, Grok Build and Hermes Agent) where repository-controlled git settings ran commands outside the agent's sandbox. Most of the detailed cases used a core.fsmonitor line in the repo's own .git/config; in Claude Code's case, the start-up git status ran it while the workspace-trust prompt was still waiting for an answer.

Both reports are about agents, but the mechanism is plain git, and git also runs in your CI jobs, bots and editor plugins. So we tested git itself: which everyday commands run programs that a repo's .git/config or hooks point at, whether the usual hardening flags stop them, and which ways of moving a repo between machines carry that config along.

TLDR

• A bare git status ran repo-supplied programs: core.fsmonitor, a post-index-change hook from .git/hooks or core.hooksPath, and on git 2.55 a hook defined in config.
• git diff --no-ext-diff --no-textconv still ran the clean filter (and a filter.<name>.process helper). Overrides only stopped a filter when we named its driver.
• On git 2.55, -c core.hooksPath=/dev/null did not stop a hook defined in config (hook.<name>.command, new in git 2.54). Adding -c hook.<event>.enabled=false did, in our fixture.
• git clone and cloning from a bundle never brought the config or hooks. A tar of the working copy, which is what a workspace cache or artifact is, always did.
• git's ownership check (safe.directory) refused a copy owned by another user on our Raspberry Pi. On GitHub-hosted runners it never fired, because the images set safe.directory = *.
• The scripts, the recorded results and a read-only audit are in the repo below.

Prerequisites

• git and bash, to run the scripts
• A machine where you can create throwaway repos: every "payload" in the tests only appends a line to a log file

Why a repo can make git run programs at all

Some git settings are commands, not values. core.fsmonitor names a helper that tells git which files changed, so big repos do not stat every file. A filter driver (filter.<name>.clean, smudge or the long-running process) rewrites file content on the way into and out of the index; that is how Git LFS works. diff.external and diff.<name>.textconv replace or preprocess diffs. Hooks are executables in .git/hooks or in the directory core.hooksPath points to, and since git 2.54 a hook can also be a command defined in config (hook.<name>.command plus hook.<name>.event, see git-hook).

Git reads all of these from the repository's own .git/config and hooks directory. That is documented, intended behaviour for a repo you created. The trouble starts when .git came from someone else.

A normal clone protects you: git does not transfer .git/config or hooks over a fetch. .gitattributes does travel, because it is a tracked file, but a filter named there does nothing unless config defines it. So the question for a DevOps team is not "can a repo do this", it is "where do we copy a .git directory instead of cloning it". GitLab's report lists the answers: "an archive, a synced folder, a CI cache, or a devcontainer build", plus a compromised agent that writes the config into a repo you cloned normally.

The test

The repo builds small throwaway repositories. Each has one committed file and the same file with a line appended, so there is always a change to show, and a .git/config (or hooks directory) that points one setting at a script whose only job is to log that it ran and pass content through:

The-DevOps-Daily/git-config-exec-check on GitHub

We tested eleven settings: core.fsmonitor; clean, smudge and process filters; a textconv driver; diff.external; hooks in .git/hooks; hooks via core.hooksPath; a hook defined in config; core.sshCommand; and core.pager. For each one, scripts/matrix.sh copies the repo fresh, runs 15 commands that tools and people run all the time, and records the exit code and what ran.

We ran it on a Raspberry Pi with git 2.39.5, and on GitHub-hosted ubuntu-latest and macos-latest runners with git 2.55.0. Ubuntu and macOS were identical. Git 2.39.5 ran the same programs except for the config-hook column, which it does not support. This is the git 2.55.0 table:

How to read it:

• A blank cell means "not observed with this fixture", not "can never run". Because our modified file is longer than the committed one, git can see the change from the file size without filtering it. With a same-size edit or a touched file, git status may need to compare content, and then filters can run too.
• hooks dir was identical for .git/hooks and core.hooksPath, so they share a column. core.pager never ran, because none of the commands had a terminal; agents and CI jobs usually do not either.
• Our process helper does not speak git's filter protocol. Git started it either way; git 2.39.5 then exited with 128, and git 2.55.0 carried on and exited 0.
• Only the sshCommand fixture has a remote, and its fetch fails on purpose. A successful fetch can fire more hooks than this row shows.

Three rows deserve a second look.

git status is not read-only. It refreshes the index, and refreshing the index is when git asks the fsmonitor helper what changed. When the refresh writes the index back, git fires the post-index-change hook, from the hooks directory and from config. All of that happened on a repo with one modified file.

git fetch ran fsmonitor before it contacted the remote. In the fsmonitor fixture there is no remote at all. scripts/trace-fetch.sh shows git starting the helper twice (asking for protocol version 2, then falling back to version 1 after our helper exited non-zero) and only then failing to find origin:

git 2.39.5 on Linux aarch64
run-command.c:655       trace: run_command: cd <tmp>/r; '<tmp>/hook.sh fsmonitor' 2 1791269076974045040
run-command.c:655       trace: run_command: cd <tmp>/r; '<tmp>/hook.sh fsmonitor' 1 1791269076974045040
run-command.c:655       trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 'git-upload-pack '\''origin'\'''
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

The diff flags that hardened tools add are aimed at the wrong half. --no-ext-diff and --no-textconv stop the programs that render a diff. The clean filter runs earlier, when git turns the working-tree file into a blob to compare. That is the GitLab finding, and plain git 2.55 behaves the same way. Diffs between two commits (git log -p, git show) never ran a filter, because no working-tree file is involved.

Do the usual flags help?

scripts/overrides.sh builds one repo with fsmonitor, a hook in .git/hooks, a hook defined in config, diff.external, a textconv driver and a clean filter. Each row starts from a fresh copy, runs git status and git diff, and adds more protection. On git 2.55.0:

git 2.55.0 on Linux x86_64
plain                                                    exit 0/0  ran: diff-external,filter-clean,fsmonitor,hook:config,hook:dir
diff --no-ext-diff --no-textconv                         exit 0/0  ran: filter-clean,fsmonitor,hook:config,hook:dir
+ -c core.fsmonitor=false -c core.hooksPath=/dev/null    exit 0/0  ran: filter-clean,hook:config
+ -c filter.lab.clean= (needs the driver name)           exit 0/0  ran: hook:config
+ -c hook.post-index-change.enabled=false (per event)    exit 0/0  ran:

Read it row by row:

1. With no protection, five programs ran: fsmonitor, the hooks-directory hook, the config hook, the clean filter and diff.external. The textconv driver did not, because diff.external replaces the whole diff; the matrix above is where textconv shows up, and where --no-textconv stops it.
2. The diff flags stopped diff.external, and nothing else.
3. core.fsmonitor=false and core.hooksPath=/dev/null are what a careful wrapper adds. They stopped fsmonitor and the hooks-directory hook, but not the hook defined in config, which is not looked up in a hooks directory. The clean filter also still ran.
4. Clearing the clean filter worked only because we knew the driver was called lab. An attacker picks the name, and .gitattributes can name a different driver for each file pattern. The same goes for smudge and process, which a wrapper has to clear too.
5. hook.<event>.enabled=false, which the git 2.55 git-hook documentation describes as a switch for every hook of one event, stopped the config hook in this fixture without knowing its name. We only tested it together with core.hooksPath=/dev/null, so keep both. It is also per event, so a wrapper has to list every event its commands can fire.

On git 2.39.5 the same script gave the same first four rows, minus the config hook, which that version does not know about.

Which copies bring the config along

scripts/delivery.sh builds one repo with core.fsmonitor, a clean filter and a post-index-change hook. All three call a helper stored inside .git by relative path, so the payload travels with any copy that includes .git. It copies the repo four ways, records anything that ran during the copy, and then runs git status and git diff in each copy. On the Pi:

git 2.39.5 on Linux aarch64
safe.directory already set on this machine: no
original repo                          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
git clone                              copy ran: nothing  then ran: nothing                                        exit 0/0
git clone from a bundle                copy ran: nothing  then ran: nothing                                        exit 0/0
tar of the working copy                copy ran: nothing  then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
same tar, owned by another user        copy ran: -        then ran: nothing                                        exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.S42grY0m9P/other-owner'
  ...with no system or global config   copy ran: -        then ran: nothing                                        exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.S42grY0m9P/other-owner'
  ...with -c safe.directory=*          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0

The clone and the bundle were clean, both during the copy and afterwards. The tar was not, and a tar of the working copy is what most "cache the workspace" setups produce: an actions/cache or GitLab cache: entry that includes .git, a build artifact someone zipped from the job directory, a devcontainer volume, a folder synced between machines. Once that copy lands, the next tool to run git status in it runs the repo's programs.

Here is that as a terminal session, from scripts/demo-restored-workspace.sh: a workspace restored from a tarball, two git commands, and our audit script at the end. The helper sits inside .git, so it arrived with the tarball, and the script empties ran.log before each git command.

restored workspace, git 2.39.5

# a workspace restored from a cache tarball, not cloned
$ git status --short
 M notes.txt
$ cat ../ran.log
ran: fsmonitor
ran: fsmonitor
$ git diff --no-ext-diff --no-textconv --stat
 notes.txt | 1 +
 1 file changed, 1 insertion(+)
$ cat ../ran.log
ran: fsmonitor
ran: fsmonitor
ran: filter-clean
ran: filter-clean
$ audit.sh .; echo "exit $?"
Repo config in . can run programs:
file:.git/config  core.fsmonitor    .git/lab-hook.sh fsmonitor
file:.git/config  filter.lab.clean  .git/lab-hook.sh filter-clean
exit 1

The ownership check, and the runners that turn it off

Since git 2.35.2 (the fix for CVE-2022-24765), git refuses to work in a repository owned by another user unless that path is allowed by safe.directory. On the Pi, that check stopped the poisoned tar as soon as another user owned it.

On GitHub-hosted runners, the same step ran everything. The script prints where safe.directory comes from, and on the Ubuntu runner (image ubuntu24 20260927.320.1) it said:

git 2.55.0 on Linux x86_64
safe.directory already set on this machine: system file:/etc/gitconfig *
original repo                          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
git clone                              copy ran: nothing  then ran: nothing                                        exit 0/0
git clone from a bundle                copy ran: nothing  then ran: nothing                                        exit 0/0
tar of the working copy                copy ran: nothing  then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
same tar, owned by another user        copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0
  ...with no system or global config   copy ran: -        then ran: nothing                                        exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.Y1kIjtRPwM/other-owner'
  ...with -c safe.directory=*          copy ran: -        then ran: filter-clean,fsmonitor,hook:post-index-change  exit 0/0

The Ubuntu image writes directory = * into /etc/gitconfig, and the macOS image (macos26 20260907.0351.1 in our run) adds it to the global config; you can see both in the Ubuntu and macOS image scripts. The Ubuntu script's comment says why: git 2.35.2 "introduces security fix that breaks action\checkout". With system and global config switched off, git 2.55 refused the copy exactly like git 2.39 on the Pi, so the behaviour is git's and the wildcard is the image's.

Two caveats keep this in proportion. The ownership check only helps when the files belong to a different user; a cache that the job restores as itself passes the check with or without a wildcard. And a refusal stops git, it does not make the repository safe. Still, on those runners the ownership check is not part of your defence, so it comes down to whether you restore .git at all.

To see what your own runners and build containers do, print every entry with its scope and file. Git only honours safe.directory from system, global and command-line config, so a repo cannot allow itself. A * in any of those means the check is off for all paths, unless a later empty entry resets the list, and removing a global * does nothing if the system config still has one:

git --no-pager config --show-scope --show-origin --get-all safe.directory

What to do

Do not restore .git from a cache or artifact

Cache what is expensive to rebuild, not the repository. .git rarely needs caching: actions/checkout fetches a single commit by default, and a fetch never carries config or hooks. If you need full history for speed, cache a bundle (git bundle create) and clone from it; in our test, a bundle clone ran nothing.

This closes one route, not cache poisoning in general. A restored node_modules or provider directory can also contain code that runs, and GitHub's own cache security guidance says to treat restored caches as untrusted input. Keep caches separated by trust level, so a pull request job cannot write what the main branch job restores.

Treat artifacts the same way. If a later job downloads an artifact that contains a .git directory and runs any git command inside it, that job trusts whoever produced the artifact.

Audit before the first git command

When you cannot avoid a copied .git (a synced folder, a devcontainer volume, a repo an agent has had write access to), read its config before you run git in it. git --no-pager config --list reads config files without running any of the programs above; keep --no-pager, because a configured pager can run when the output goes to a terminal.

Our scripts/audit.sh wraps that. It lists repo-level keys whose value is a command, a hooks directory or an include (including hook.<name>.command, process filters, shell aliases and pager.<cmd>), plus executable hooks in the repo's hooks directory, and exits 0 (none), 1 (found) or 2 (it could not inspect the repo, for example because git refused it). scripts/audit-selftest.sh checks it against every fixture plus odd layouts, such as a driver name containing =, a symlinked hook, a linked worktree and an include outside .git, and confirms that auditing ran nothing. It is a detector for the settings it knows, not a safety certificate.

Run a copy you trust, kept outside the directory you are checking; a poisoned workspace can replace any script inside it. In CI, that means fetching the audit at a pinned commit before you restore anything:

# before restoring the cache: fetch the audit from a pinned commit, outside the workspace
curl -fsSL -o "$RUNNER_TEMP/audit.sh" \
  "https://raw.githubusercontent.com/The-DevOps-Daily/git-config-exec-check/<commit-sha>/scripts/audit.sh" || exit 1
# after restoring it
bash "$RUNNER_TEMP/audit.sh" "$GITHUB_WORKSPACE" || { echo "restored repo can run programs"; exit 1; }

Expect some findings on developer machines. git lfs install --local writes a filter driver into the repo config, and many teams use hooks on purpose. That is fine: the point is to see them before git runs them, in a place where you did not put them yourself.

If you write tools that shell out to git

GitLab's advice is to override every relevant key on every call, or avoid git's filter and textconv machinery. From our results, for the commands a tool runs to read state (status, diff, blame, log), that means:

• Pass -c core.fsmonitor=false -c core.hooksPath=/dev/null on every call, including git status.
• On git 2.55, also pass -c hook.<event>.enabled=false for each event your commands can fire. Our table saw pre-commit, post-commit, post-checkout, post-index-change and reference-transaction.
• Add --no-ext-diff --no-textconv to diffs, and --no-pager to anything that might reach a terminal.
• For filters, read the config first and clear clean, smudge and process for every driver it defines, or use commit-to-commit diffs when they are enough.
• For fetch and push, the remote URL and core.sshCommand both come from the copied config. Set the transport yourself (for example -c core.sshCommand=ssh and an explicit URL) or do not let the tool talk to remotes from that copy.
• When the repo came from outside, run git as a user that does not own it, with no safe.directory wildcard in system or global config.

These cover what we tested. They are not a complete list of every setting git can turn into a command, which is why the audit and the "do not copy .git" advice come first.

If you use one of the agents named in the two reports, update it. The DeepSeek-Reasonix fix is Studio 2.21.0 and npm 1.39.3. Manifold's post has a table of fixed versions; four of its eight findings were still unpatched when it was published.

What we did not test

• Windows, credential helpers and editors. Each of those needs a different trigger.
• A fetch that succeeds, and a process filter that speaks the protocol. Both can run more than our rows show.
• Any specific agent. These scripts test git. How a given agent calls git decides which rows of the table apply to it, and the two reports above are the place to look for that.
• Every command and every state. We picked 15 common commands and one kind of change. git rev-parse and git log --oneline ran nothing here; that is not a promise about other commands or other repo states, so run matrix.sh with the commands your tools use.

If you have read our posts on pre-commit hook security or the MCP design flaw, this is the same lesson from another side. The dangerous input is not always code you run on purpose. Sometimes it is the configuration of the tool you run, read from the directory you are standing in.

We wanted to know how far off the obvious guesses are, so we measured. Three features on one key, two models on DigitalOcean Serverless Inference, 151 recorded calls, and every number below comes from the usage block the provider returned. The feature we expected to be expensive was not the problem. The one-word classifier was.

TLDR

• On Kimi K2.6, an incident answer the engineer read was about 100 tokens long. The request billed about 2,900 tokens, and 928 of them were reasoning tokens that nobody reads.
• A one-word alert label used a median of 566.5 reasoning tokens on Kimi K2.6 and 184.5 on DeepSeek V4.1 Flash, with single calls as high as 1,590 and 1,783. On mean costs, one label cost between 32 and 48 percent of a full incident investigation.
• Turning reasoning off for the label (reasoning_effort: "none" on Kimi K2.6) cut the cost per 1,000 labels 41 times at list price. It also changed 6 of the 20 labels.
• The agent loop was not the multiplier we expected. In 19 of 20 runs the model asked for all four tools in one step, so an investigation took two calls, not the ten we had feared.
• The same prompt used anywhere from 551 to 1,652 reasoning tokens across ten runs. Per-request cost is a distribution, not a number.
• The fix is one wrapper: every call carries the feature and tenant that caused it, and writes the provider's usage in OpenTelemetry GenAI attribute names.

Prerequisites

• Node.js 20 or later
• An API key for an OpenAI-compatible endpoint that returns a usage block (we used DigitalOcean Serverless Inference)
• A rough idea of per-token pricing: you pay one rate for input tokens and a higher rate for output tokens

The scripts, the fixtures, every raw response's usage and the report are in the companion repo:

The-DevOps-Daily/ai-feature-cost-ledger on GitHub

The invoice has the wrong shape

A cloud bill has the same problem, and the cloud answer is cost allocation tags. Model spend has no tags. The provider sees an API key and a model name, so that is what it bills by. Everything you would want to group by, such as the feature, the customer and the environment, exists only in your code at the moment you make the call.

There is a second problem, and it is the one that surprised us. Even per request, what a user sees has little to do with what you pay for. A request bills four things:

• Input tokens: the system prompt, the conversation so far, tool definitions and tool results, every time
• Cached input tokens: the part of the input the provider served from its prompt cache, at a lower rate
• Output tokens: the answer
• Reasoning tokens: thinking the model does before it answers, billed as output and usually never shown

The first and last are the ones people underestimate. So we built three small features that share one key and measured each.

The three features

All three run on the same key, which is the point: the provider sees one stream of calls.

1. Incident assistant. An engineer asks why checkout-api is returning 502s. The assistant has four tools: list pods, read logs, list recent deploys and read metrics. The tools return fixed text describing pods that are OOM-killed after a deploy raised a cache size, so every run sees the same incident. We ran it two ways: as a tool-calling agent, and single-shot with all four tool outputs pasted into one prompt. The system prompt asks for at most three short sentences.
2. Ticket summary. Two sentences for an account manager from a nine-message support thread.
3. Alert classifier. One word, page, ticket or ignore, for 20 different alert lines.

Each incident condition ran 10 times on each of two models, DeepSeek V4.1 Flash and Kimi K2.6. The classifier ran all 20 alerts under four settings. Both models report reasoning tokens and cached tokens separately in usage, which is why we picked them. All 40 incident answers identified the intended root cause (a keyword check for the memory exhaustion and the cache change behind it), so the cost comparisons below are between answers that found the right cause.

Prices are DigitalOcean's published Standard rates per million tokens, read from the pricing page on 5 October 2026:

What one answer actually billed

Here is the single-shot incident answer on Kimi K2.6, straight from the report file:

ai-feature-cost-ledger

# one incident answer on Kimi K2.6, median of 10 runs
$ jq '.incident["kimi-k2.6 single"] | {medianInput, medianOutput, medianReasoning, medianVisibleAnswer, medianBilledPerVisible}' data/report.json
{
  "medianInput": 1893,
  "medianOutput": 1023.5,
  "medianReasoning": 927.5,
  "medianVisibleAnswer": 102,
  "medianBilledPerVisible": 30.6
}

The engineer read a 102-token answer. The request billed a median of 1,893 input tokens and 1,024 output tokens, and 928 of the output tokens were reasoning. For every token the engineer read, the request billed about 30 (the median of the ten per-run ratios). On DeepSeek V4.1 Flash the same question billed about 17 tokens per token read, because it reasoned less: a median of 126 reasoning tokens against Kimi's 928.

One incident answer: what was billed vs what was read

Kimi K2.6, single-shot, median of 10 runs. Each bar is its own median, so the bars need not add up. Source: data/report.json in the companion repo.

Warning

Reasoning tokens are in completion_tokens, and you pay the output rate for them. If you estimate cost by counting the tokens in the answer text, you miss most of the output bill on a reasoning model. Read usage.completion_tokens_details.reasoning_tokens where the provider reports it.

The agent loop was not the problem

We expected the agent to be the expensive version. An agent resends the whole conversation on every step: system prompt, tool definitions, every earlier tool result. Ten sequential steps means paying for the first prompt ten times.

That is not what happened. In 19 of the 20 agent runs, the model asked for all four tools in its first step, received the results, and answered in the second. The one exception, a Kimi run, asked for three tools, then the fourth, then answered: three calls. With two calls, the resend costs one extra copy of a short prompt.

Those are list prices with no cache discount, which is the fair comparison here (the cache section explains why). On Kimi the agent was no dearer at all, because it reasoned less once the tool results were in front of it.

This does not mean agent loops are cheap. It means the step count decides, and the step count belongs to the model and the task, not to you. A model that calls one tool per step, or a task where each tool result decides the next call, turns two calls into eight. Measure steps per request as its own number, and alert on it.

The one-word answer that cost the most

The classifier is the feature nobody worries about. The prompt is roughly 50 to 90 tokens and the answer is one word. At list price, it cost $0.48 per 1,000 labels on DeepSeek V4.1 Flash with default settings, and $2.45 on Kimi K2.6.

Almost all of that is reasoning. The answer was 2 to 4 tokens every time. The reasoning was not:

Reasoning tokens spent on a one-word alert label

20 alerts per setting, one call each. The visible answer was 2 to 4 tokens every time. Source: data/feature-runs.jsonl.

The tail is where the money goes. A DNS failure rate of 14 percent for three minutes drew 1,783 reasoning tokens from DeepSeek before it said ticket. A node memory-pressure alert drew 1,218. On DeepSeek at default settings, 14 of the 20 alerts used between 50 and 250. Nothing in the request tells you in advance which call will be the expensive one.

Put next to the incident assistant, this is the number that changes priorities. Comparing mean costs, one label cost 32 to 48 percent of a full agent investigation: 32 percent on DeepSeek at list price, 48 percent on Kimi with the cache discounts both features got. A feature that runs on every alert is billed at a third to a half of a feature that runs when an engineer is paged.

Turning reasoning down, and what it changes

Both models accept a reasoning_effort parameter on this endpoint, but not the same values, and the values do not mean the same thing. DeepSeek V4.1 Flash rejected none and minimal (reasoning_effort must be one of [low high xhigh max] for this model) and accepted low. Kimi K2.6 accepted all three, but on the probe alert low and minimal used more reasoning than the default (1,210 and 1,021 tokens against 656), and only none turned it off. That is one call each, recorded in data/effort-probe.json, so read it as "check what the setting does", not as a rule.

ai-feature-cost-ledger

# the alert classifier on Kimi K2.6, default vs reasoning_effort none
$ jq '.classifier["kimi default"] | {runs, medianOutput, medianReasoning, maxOutput, medianVisible, costPer1000}' data/report.json
{
  "runs": 20,
  "medianOutput": 569.5,
  "medianReasoning": 566.5,
  "maxOutput": 1593,
  "medianVisible": 3,
  "costPer1000": 2.4195
}
$ jq '.classifier["kimi effort=none"] | {runs, medianOutput, medianReasoning, costPer1000, agreesWithDefault}' data/report.json
{
  "runs": 20,
  "medianOutput": 2,
  "medianReasoning": 0,
  "costPer1000": 0.0381,
  "agreesWithDefault": 14
}

Cost per 1,000 alert labels

Mean cost of 20 real calls per setting, scaled to 1,000, at DigitalOcean list prices on 5 Oct 2026 with no cache discount (costPer1000NoCache). Source: data/report.json.

With reasoning off, Kimi produced 2 output tokens per label. At list price, the cost per 1,000 labels fell from $2.45 to $0.06, 41 times less. As billed, with the cache discounts some of those calls got, it fell from $2.42 to $0.04, 63 times less. DeepSeek at low saved 1.8 times either way, and its tail did not go away: one call still used 1,138 reasoning tokens.

The catch is in the last field. Without reasoning, Kimi gave a different label on 6 of the 20 alerts. Four moved up to page: the disk at 91 percent, a node under memory pressure, a CI queue delay and a single OOM-killed pod. The other two moved from ignore to ticket. DeepSeek at low changed 2 labels. Each alert ran once per setting, so these are observed differences, not a measured rate, and we have no ground truth for the alerts, so we cannot say which setting was right. We can say that "turn reasoning off" is a product decision about who gets woken up, not a free saving. Run your own labelled set through both settings before you flip it.

Caching is a discount you cannot schedule

Prompt caching is the other big lever. DeepSeek V4.1 Flash bills cached input at $0.006 per million tokens against $0.30 uncached, 50 times less. On the nine single-shot runs that hit the cache, the median cost was $0.000342, against $0.000925 for the same tokens at list price.

Two things stop you from budgeting on that number:

• The first call always paid full price. The first call of every incident condition had zero cached tokens, and so did the first call of the second Kimi agent run, even though the run before it had just sent the same prefix.
• What repeats in our test does not repeat in production. We sent the identical incident ten times, so later runs found almost the entire single-shot prompt in cache (1,984 of 2,019 tokens on DeepSeek). A real incident brings new logs and new metrics. Only the shared prefix (system prompt and tool definitions, 576 tokens on DeepSeek here) can repeat across different incidents, which is why the agent comparison above uses list prices.

Record cache_read tokens per call and watch the hit rate as its own metric. Put the stable part of every prompt first and the per-request part last, so the prefix the cache can match is as long as possible. Then budget at list price and treat the cache as a discount, not a plan.

Tag every call with the feature that caused it

None of the numbers above helps finance unless each call says which feature it belongs to. The fix is boring, and that is the point. Every model call goes through one function. The function knows the feature and the tenant, reads the usage block from the response, prices it, and writes one row:

// src/ledger.mjs, trimmed
import { appendFileSync } from 'node:fs';
import { chat } from './client.mjs';
import { costUsd, splitUsage } from './cost.mjs';

export async function meteredChat({ feature, tenant, ledger }, body) {
  const res = await chat(body);
  const s = splitUsage(res.data.usage);
  const row = {
    ts: new Date().toISOString(),
    'app.feature': feature, // who caused this call
    'app.tenant': tenant, // who you would bill or rate limit
    'gen_ai.operation.name': 'chat',
    'gen_ai.provider.name': 'digitalocean',
    'gen_ai.request.model': body.model,
    'gen_ai.response.model': res.data.model,
    'gen_ai.usage.input_tokens': s.input, // includes cached tokens
    'gen_ai.usage.cache_read.input_tokens': s.cached,
    'gen_ai.usage.output_tokens': s.output, // includes reasoning tokens
    'gen_ai.usage.reasoning.output_tokens': s.reasoning,
    'app.cost_usd': costUsd(body.model, res.data.usage),
  };
  appendFileSync(ledger, JSON.stringify(row) + '\n');
  return res;
}

And the pricing function it calls, which is where most homemade cost trackers go wrong:

// src/cost.mjs, trimmed
export function splitUsage(u = {}) {
  const input = u.prompt_tokens ?? 0;
  const cached = u.prompt_tokens_details?.cached_tokens ?? u.cache_read_input_tokens ?? 0;
  const output = u.completion_tokens ?? 0;
  const reasoning = u.completion_tokens_details?.reasoning_tokens ?? 0;
  return { input, cached, uncached: input - cached, output, reasoning };
}

export function costUsd(model, usage) {
  const p = pricing.models[model]; // USD per million tokens, with source and date
  const s = splitUsage(usage);
  const cachedRate = p.cachedInput ?? p.input;
  return (s.uncached * p.input + s.cached * cachedRate + s.output * p.output) / 1e6;
}

Three details matter more than the rest.

• Use the provider's numbers, not your own estimate. The usage block is what you are billed for. A tokenizer on your side counts the text you sent; it does not see reasoning tokens or what the cache served.
• Split before you price. prompt_tokens already includes cached tokens and completion_tokens already includes reasoning tokens. Add them on top and you bill yourself twice for the same tokens.
• Name the fields the way OpenTelemetry does. The GenAI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.reasoning.output_tokens, and state that the cached and reasoning counts are included in the totals, the same rule as above. The conventions are still marked Development, so pin the version you emit. The names still mean a JSON line today can become a span attribute later without a rename.

Once every row carries app.feature, the roll-up is a GROUP BY. The 151 calls in this experiment cost $0.19 as billed and $0.22 at list price. As billed, the 80 classifier calls cost more than the 41 agent calls; at list price, the agent calls were slightly ahead:

The mix of calls here is our test plan, not a production workload, so the shares mean nothing on their own. The shape is the lesson: the feature with the shortest answers spent the most reasoning tokens.

Where a gateway fits

You can stop at the wrapper. It is about 30 lines, it lives in your code, and it adds no one who sees your prompts beyond the inference provider. Keeping it consistent gets harder when the calls come from many services in several languages, and it reports spend without enforcing a budget. That is the problem the AI gateway and LLM observability vendors sell into, and they solve the tagging part in similar ways:

• Portkey takes an x-portkey-metadata header with any keys you choose, such as feature and _user, and has budget limits in front of the provider.
• Helicone uses one header per property, Helicone-Property-<Name>, and lets you segment requests and cost by them.
• Langfuse records usage and cost per generation. It infers cost from model definitions it ships for OpenAI, Anthropic and Google models, so for a model like DeepSeek V4.1 Flash on DigitalOcean you add your own definition with the prices, or send usage and cost yourself. Ingested values take priority over inferred ones, and you should send them.
• Braintrust logs traces with token counts, cost and metadata alongside its evaluations, which fits the question this post leaves open: whether a cheaper reasoning setting still labels alerts correctly.

The trade-offs are the usual ones. A proxy gateway adds a network hop to every model call, and a hosted one is another company that sees your prompts. Helicone, Langfuse and Portkey's gateway all publish source you can run yourself, which answers the second point and makes the first your problem. Whichever you pick, check that it records the provider's reported usage, including reasoning and cached tokens, rather than estimating from the text. Counting only what the user saw would have put the incident assistant's token bill far too low in this test: the median billed-to-read ratio per condition ran from 17 to 40, and single investigations from 14 to 50.

What we could not conclude

• Whether failed attempts are billed. The 151 recorded calls all succeeded on the first attempt, so the client's retries for 429 and 5xx never fired in the record. A request that fails or times out on the client side may still reach the provider and be billed, and nothing we recorded can show whether it was.
• Which classifier labels were right. We have no ground truth for the 20 alerts, so the 6 changed labels are a difference, not an error rate.
• Whether reasoning helps the incident answers. All 40 identified the intended root cause by our keyword check, which is not a full accuracy review, so we cannot price the accuracy reasoning buys on this task. A harder incident might change that.
• Anything about other models. Two models, one endpoint, one day. Others reason more or less, and the OpenAI GPT-5 models on the same pricing page were not available on our account tier, so we could not include them.
• Steady-state cache hit rates. Our repeated prompts overstate hits for anything that changes per request.

Summary

• Price every request from the provider's usage block. The answer text is not the bill.
• On reasoning models, reasoning tokens can be most of the output bill, even for a one-word answer. Track reasoning_tokens as its own metric and look at the tail, not the median.
• Steps per request decides what an agent costs. The median was 2 here: 19 investigations took two calls and one took three. Measure it and alert when it climbs.
• reasoning_effort is a large lever and a product decision. Test the labels it changes before you ship it.
• Budget at list price. Treat the prompt cache as a discount, put the stable prefix first, and watch the hit rate.
• Wrap every model call once, tag it with the feature and tenant, and use the OpenTelemetry GenAI names. Then finance gets one line per feature, and you get it before they ask.

For two other ways a model bill grows quietly, see why most LLM calls are classification and what a semantic cache really saves.

Most of them missed. The latest gitleaks release, with its default rules, found nothing. The common hand-written patterns hid only the first 40 to 46 characters, and in our sample that part never changed from one token to the next, so the "redacted" log line still held everything an attacker needs.

TLDR

• The Actions token we measured is 377 characters, not the "about 520" in GitHub's changelog. All 40 samples had the same length and shape: ghs_15368_ plus a three-part JWT signed with ES256.
• The first 46 characters were identical in all 40 tokens. They are the app ID of GitHub Actions and a base64url JWT header that never changed. A rule that redacts only that part hides nothing secret.
• Five common patterns redacted the whole token in 0 of 40 cases. Three never matched at all, so the full token would stay in the log. Two matched only the 40 or 46 public characters.
• GitHub's own first recommended regex left the hyphen out. The May 15 version, ghs_[A-Za-z0-9\._]{36,}, fully matched 8 of 40 tokens, because 32 had a - in the JWT. GitHub corrected it on May 26, and the fixed regex matched all 40.
• gitleaks 8.30.1 (latest release, March 21) reported 0 findings. trufflehog 3.97.9 found a 46-character fragment and marked it unverified. detect-secrets 1.5.0 flagged the line. Fixes for gitleaks and trufflehog have been open pull requests since July.
• Storage fails loudly, except one case. Postgres and default MySQL 8.4 reject the token in a VARCHAR(255) column. MySQL with strict mode off stores the first 255 characters with only a warning.
• Two custom rules, gitleaks and trufflehog, catch the full 377-character token today. Both are below and in the companion repo.

Prerequisites

• Code, logs, or a database that handles GitHub App installation tokens or the Actions GITHUB_TOKEN
• ripgrep (rg) to search a codebase for old patterns
• Optional: a GitHub account to fork the test repo and run it against your own rules

What changed, and when

GitHub announced the change on April 24. The staged rollout started on April 27 with the Actions GITHUB_TOKEN and first-party integrations, then moved to all App installation tokens from mid-May to late June. The October 2 post marks it as complete.

What changed:

• Format: ghs_APPID_JWT. The ghs_ prefix stays.
• Length: "~520 characters" in GitHub's words, and it "will vary based on the data stored within it."
• Contents: GitHub says the JWT "contains details about the token such as the target installation, the application, and basic validation details," and that clients must not validate it or depend on its contents.

What did not change: permissions, repository scoping, the one-hour expiration, and the REST endpoint that mints the token. GitHub Enterprise Server is not affected.

One date still matters. GitHub added a temporary X-GitHub-Stateless-S2S-Token request header in May so apps could force either format. Apps that send disabled to keep getting old tokens lose that option on November 30, 2026, when GitHub stops respecting the header.

How we tested without leaking a token

We did not have a third-party GitHub App to mint tokens from, so we used the token every Actions job already has. GitHub's April notice says the new format covers "GitHub App installation server-to-server tokens, including Actions GITHUB_TOKEN." A workflow in a private test repository read secrets.GITHUB_TOKEN and passed it to small Python scripts that print only structure: lengths, separator positions, the decoded JWT header, the claim names, and how many characters each regex leaves visible. No step printed the payload values or the signature, and we checked the downloaded logs for any token value before making the repository public.

Two workflows ran on October 5, 2026, on ubuntu-latest:

• check.yml: one token through the shape script, the regex list, gitleaks, trufflehog, detect-secrets, and Postgres and MySQL service containers.
• sample.yml: 40 matrix jobs, one token each, to see whether the results hold across tokens.

This is the shape script's output from the recorded check run:

check.yml, Token shape step

# TOKEN is the job's GITHUB_TOKEN; the script prints structure only
$ python3 scripts/shape.py
length: 377
prefix: ghs_
separators (index, char): [(3, '_'), (9, '_'), (46, '.'), (290, '.'), (321, '-'), (360, '-')]
app id segment: 15368 (public: the app's numeric id)
segment lengths header/payload/signature: 36 243 86
chars used outside [A-Za-z0-9]: ['-']
header: {"alg":"ES256","typ":"JWT"}
payload claim names: ['aud', 'ctx', 'exp', 'iat', 'iss', 'jti', 'ver']
exp - iat (s): 3600
signature bytes: 64

Put together, a current GITHUB_TOKEN looks like this:

ghs_15368_eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.<243-character payload>.<86-character signature>
|____________________________________________|
  46 characters, the same in all 40 samples:
  ghs_ + 15368 (the app ID of GitHub Actions) + _ +
  the base64url of {"alg":"ES256","typ":"JWT"}

A few things we did not expect:

• 377, not 520. All 40 Actions tokens were exactly 377 characters. GitHub's "about 520" probably describes tokens for other apps, which carry more claims. We could not mint one, so we cannot confirm that. Plan for at least 520, as GitHub asks, and do not assume a fixed length either way.
• The prefix is public. The 46-character head (ghs_15368_ plus the header segment) had one SHA-256 value across all 40 samples. It decodes to {"alg":"ES256","typ":"JWT"}, which anyone can encode.
• The JWT uses base64url, so - and _ appear inside it. 32 of the 40 tokens had a -, 27 had a _, and only 3 had neither. This detail breaks the most regexes.
• Actions masks the token in its own logs, but nobody else does. GitHub masks secret values in workflow logs and says that this redaction is not guaranteed. Your application logs, proxies, error trackers, and log pipelines do not get that masking at all.

The redaction test

For each pattern, the scripts ran re.search against the token and counted the characters outside the first match. Every pattern here starts with ghs_, which appears once per token, so that is also what a re.sub replacement would leave visible. The patterns are ones you find in real redaction code: GitHub's old documented shape, the regexes from the current gitleaks and trufflehog releases, the two regexes GitHub recommended in May, and the regexes from the open scanner fixes.

Tokens fully redacted, out of 40 real GITHUB_TOKENs

Each pattern run with Python re against 40 Actions tokens minted on 2026-10-05 (sample.yml in The-DevOps-Daily/ghs-token-check). 'Fully redacted' means re.sub would replace all 377 characters.

The failures fall into three groups.

Never matches, so the full token stays in the log. The legacy shape and the gitleaks rule expect 36 letters or digits right after ghs_. The new token has five digits (15368) and then an underscore, so the match fails at character 10. The word-bounded variant allows underscores, but its 36 characters end in the middle of the header, where no word boundary exists.

Matches the public prefix only. Allow underscores and the pattern matches ghs_15368_ plus 30 header characters. Make it open-ended ({36,255}, which is what trufflehog uses) and it runs to the first . and stops at 46 characters. Either way, the payload and the full signature stay in the log line. In our sample the part that disappeared was the same in every token, so anyone who reads the redacted line can paste the prefix back. The result is a working token for as long as it lives, which for an installation token is up to an hour.

Stops at the first hyphen. GitHub's May 15 changelog recommended ghs_[A-Za-z0-9\._]{36,} "to match both new and current format tokens." A snapshot from May 19 shows that version. An editor's note dated May 26 updated it to ghs_[A-Za-z0-9\.\-_]{36,}. The first version has no -, and base64url uses -. In our sample, the match stopped somewhere in the signature on 32 of 40 tokens and left 8 to 86 characters visible. For redaction this is the least bad failure, because the hidden part always includes the payload, and in most of those 32 tokens part of the signature too. In one sample the whole signature stayed visible. For validation it is worse: code that checks a token with fullmatch against the May 15 regex rejects every token that contains a hyphen. In our sample that was 80% of real tokens.

If you copied GitHub's regex in the second half of May, check which version you have.

What the scanners found

The check workflow wrote the token into a file as GITHUB_TOKEN=<token> and pointed the latest release of each scanner at it:

gitleaks v8.30.1
gitleaks findings: 0
trufflehog 3.97.9
trufflehog findings: 1
  Github verified False raw_len 46
detect-secrets 1.5.0
detect-secrets findings: 2
  JSON Web Token
  GitHub Token

• gitleaks 8.30.1 found nothing. Its github-app-token rule is (?:ghu|ghs)_[0-9a-zA-Z]{36}, the first "never matches" case above. Someone reported the format change in issue #2192 on July 15, and PR #2193 with a fix has been open since July 16. The latest release is from March 21. If gitleaks 8.30.1 with its default rules guards your pre-commit hook or CI, its rule cannot match this token shape, and it did not flag our test file.
• trufflehog 3.97.9 found a fragment. Its detector matched the 46-character prefix and reported that fragment as an unverified finding. Most examples in trufflehog's README run with --results=verified, and that filter drops this finding. PR #5156, open since July 27, adds a pattern for the new format.
• detect-secrets 1.5.0 flagged the line twice, once as a JSON Web Token and once as a GitHub Token. Its GitHub pattern also matches only a prefix, but a scanner only needs to flag the line, and it did.

That last point explains why the same partial match is fine in one tool and bad in another. A scanner that matches 46 characters still points a person at the right line. A redactor that matches 46 characters prints the rest of the token.

Storing the token

GitHub asks for columns that "accept at least 520 characters." We inserted the 377-character token into narrow columns to see how each database reacts:

postgres t40:
ERROR:  value too long for type character varying(40)
postgres t255:
ERROR:  value too long for type character varying(255)
mysql 8.4 default sql_mode: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
mysql strict t255:
ERROR 1406 (22001) at line 1: Data too long for column 'tok' at row 1
mysql non-strict t255:
stored_len
255

Postgres and MySQL 8.4 with its default strict mode fail on insert, which is the good outcome: the error is in your logs the first time a new token arrives. MySQL with sql_mode cleared, which some older applications set on purpose, stores the first 255 characters and keeps going. The insert succeeds with a warning that is easy to miss, and the failure shows up later as a 401 from the GitHub API.

Installation tokens last an hour, so most apps cache them rather than store them for long. Look anyway: caches with fixed-size keys or values, cookie-backed sessions, and old migrations from when the token was always 40 characters.

What to change

1. Find the old patterns. Search code and config for anything that encodes the old shape:

# regexes that expect the old 36-character body, and anything mentioning ghs_
rg -n --hidden --glob '!.git' --glob '!node_modules' -e 'ghs_' -e 'gh\[[a-z]+\]_' -e '\{36\}' .

# columns that might hold a token and are too short
rg -n -i -e 'varchar\((40|64|100|128|255)\)' --glob '*.sql' --glob '*migration*' .

If your log pipeline or error tracker has its own scrubbing rules, check those too. A rule written for the old shape lives wherever someone pasted it.

2. Redact with a pattern that covers the whole JWT. GitHub's corrected regex works:

import re

# GitHub's recommendation as of May 26, 2026; matches old and new ghs_ tokens
GHS = re.compile(r"ghs_[A-Za-z0-9\.\-_]{36,}")

def redact(line: str) -> str:
    return GHS.sub("ghs_[REDACTED]", line)

It is greedy, so it also eats a trailing . or - that ends a sentence. For redaction that is the right tradeoff.

3. Do not validate the structure. GitHub says clients "must not take a dependency on the contents of this JWT." If you check tokens at all, check the ghs_ prefix and a generous maximum length, and let the GitHub API decide if the token is valid.

4. Patch your scanners until the upstream fixes ship. Both of these caught the full 377-character token in the recorded run (gitleaks with config/gitleaks.toml findings: 1, secret_len 377 and CustomRegex verified False raw_len 377):

Custom rules for the new installation token format

gitleaks

# .gitleaks.toml: keep the default rules, add one
[extend]
useDefault = true

[[rules]]
id = "github-app-installation-token-jwt"
description = "GitHub App installation token, stateless ghs_<app id>_<JWT> format"
regex = '''ghs_[0-9]+_eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'''
keywords = ["ghs_"]

trufflehog

# trufflehog --config trufflehog.yaml
detectors:
  - name: GitHubInstallationTokenJWT
    keywords:
      - ghs_
    regex:
      token: 'ghs_[0-9]+_eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'

A trufflehog custom detector without a verification endpoint reports its findings as unverified. A run with --results=verified drops them too, so let unverified results through for this detector.

5. Size storage for the documented length, not the one you measured. Use text in Postgres, or at least VARCHAR(1024) where you need a bound, and make sure MySQL runs in strict mode so a short column fails on insert.

6. Remove the override header. If your app sends X-GitHub-Stateless-S2S-Token: disabled to keep old-format tokens, that stops working on November 30, 2026. Fix whatever needed it before then.

Test your own rules

The workflows, scripts, and recorded results are public. Fork the repository, add your redaction or validation regex to patterns.txt, and run the workflows. The test uses the fork's own GITHUB_TOKEN, prints no token, and reports how much of a live token your rule leaves visible.

The-DevOps-Daily/ghs-token-check on GitHub

What we could not test

• Tokens from other GitHub Apps. We measured only the Actions GITHUB_TOKEN. GitHub says other installation tokens are about 520 characters, and their payloads may differ. Every regex that covers the full JWT structure should still match, but we did not run one.
• GitHub's own secret scanning and push protection. Testing them means pushing a live token to a repository, which we chose not to do, and it would say little about your own pipeline.
• Commercial log scrubbers. We tested regexes and three open source scanners, not the built-in rules in Datadog, Splunk, or Sentry. Run your own provider's rule through the test repo if you can export it as a regex.
• How long the prefix stays fixed. In our 40 samples the header segment never changed. GitHub could change the signing algorithm or add a header field at any time, so treat "the first 46 characters are public" as a property of today's tokens, not a guarantee.

Summary

GitHub warned about this change for five months and lists "logging and secret redaction rules" in its own checklist. The tools many teams use for that job have not caught up: the latest gitleaks release does not detect the new tokens, trufflehog sees only a fragment that it cannot verify, and the common hand-written patterns redact a prefix that, in our tests, was identical in every token. The fixes are small. Use a pattern that covers the whole JWT, add one custom rule to each scanner, give the column room for 520 characters or more, and test the result against a real token instead of a 40-character example in a unit test.

If you are reviewing the rest of your GitHub Actions setup, our posts on self-hosted runner version enforcement and on finding secrets before you have to rotate them cover two related changes.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!