On this page
- Which secrets do I have to rotate?
- What do I rotate first?
- Why is changing the variable in the dashboard not rotation?
- Should I revoke first or create first?
- What did each October 2026 breach tell owners to do?
- What is the runbook, step by step?
- What if the breached thing ran code on my machine?
- What does rotation not fix?
- What do I ask the host or vendor?
- What can a scan add?
To rotate secrets after a breach, list every credential the breached system could read, then for each one revoke it with the service that issued it, issue a replacement, redeploy and restart whatever uses it, and check the old one now fails. Do this broadest key first. Editing a variable in your host's dashboard is only the middle step.
| Fact | Detail | Source |
|---|---|---|
| Railway | Disclosed 21 October 2026 that environment variables may have been accessed and some decrypted; asked all users to roll every variable | Railway disclosure |
| Railway tokens | All API and project tokens invalidated 22 October 2026 at 04:39 UTC; new ones needed | Railway status |
| PingCAP / TiDB Cloud | Intrusion 9 September to 7 October 2026; report 19 October; customer actions 20 October | PingCAP report |
| Zeabur | Second incident 14 to 15 October 2026; preliminary report 19 October | Zeabur report |
| LiteLLM 1.107.0 | Malicious artifacts live on 21 October 2026, PyPI from 11:39 to 13:23 UTC | LiteLLM report |
| Checked | 24 October 2026. All four reports are preliminary or vendor-reported | this page |
This is the runbook for any host or tool breach, with the October 2026 cases as examples. The running list of incidents is in the October incident roundup; the short version is a one-week action list. More like this is in the other secrets and supply-chain guides.
Which secrets do I have to rotate?
Every secret the breached system could read, and every secret a breached secret could create.
Railway's disclosure names API keys, database and registry credentials, and there is a Railway-only walkthrough. Zeabur adds signing and encryption keys; what Zeabur customers should do now is on its own page. LiteLLM adds the cloud role of the host and any secret in the environment of any process on it, with a separate page for anyone who ran version 1.107.0. TiDB Cloud users can use the TiDB-specific rotation list.
The test is simple: did you paste the value into the breached platform, or could a process on the breached machine read it? If yes, it is in scope whether or not the provider has emailed you. Railway says a vendor's revocation notice does not by itself mean your credentials were affected (preliminary report). Its status page adds that workspaces outside the groups it had notified were not ruled out (23 October 2026).
What do I rotate first?
Broadest first. This ordering is our judgment, built on the principle in OWASP's secrets management cheat sheet that you revoke and rotate first and clean up after.
| Order | What | Why it goes here |
|---|---|---|
| 1 | Cloud account keys and roles, platform API tokens, CI tokens, GitHub tokens | They can create or read the rest. Railway invalidated all its API and project tokens on 22 October |
| 2 | Database passwords | They guard the data that cannot be un-read |
| 3 | Paid third-party keys: payments, AI providers, email and SMS | Misuse costs money fast; providers show usage by key |
| 4 | Signing, session and encryption keys | Rotating them signs users out or needs re-encryption, so plan it |
| 5 | Webhook secrets and low-privilege tokens | Real, but rarely the first thing used |
Why is changing the variable in the dashboard not rotation?
A secret lives in three places: the service that issued it, the platform variable that stores it, and the running process that loaded it. Rotation means all three change. A dashboard edit changes the middle one.
Three vendor reports describe the gap. Zeabur says changing the environment variable does not change the database password, so you change the password inside the database. Railway says regenerating a PostgreSQL credential restarts the database, and every dependent service keeps the old password until it is redeployed. Railway also removed stored GitHub registry tokens from services and says to revoke the old token at GitHub too, since it may have been exposed.
Should I revoke first or create first?
It depends on whether you think the old key was read. The sources differ on order because they answer different situations.
| Situation | Order | What the sources say |
|---|---|---|
| The old key may be in hostile hands | Revoke first, then redeploy the new one. Accept a short outage | Railway's guide for OpenAI keys revokes the old key, then updates the service. Stripe's Dashboard lets you set the old key to expire Now |
| No sign the old key was read, and downtime is costly | Create the replacement, switch the app, then revoke | GitHub: create a replacement with the same permissions, switch the app, revoke the old secret. Stripe overlaps the two for up to 7 days |
| Cloud service account keys | Create, deploy, disable the old key, then delete | Google Cloud advises disabling a key before deleting it, and deleting only once you are sure it is unused |
In a breach, the first row is the default. An overlap window is time in which the old key still works for whoever holds it.
What did each October 2026 breach tell owners to do?
Each report is preliminary and written by the vendor. They show what customers were told to do, and where the gotchas are.
| Platform | What to rotate | Gotcha | Source (date) |
|---|---|---|---|
| Railway | Every environment variable stored there; private registry credentials; API and project tokens (invalidated 22 Oct) | Redeploy needed to pick up rotated bucket credentials; redeploying code from before the incident fails until GitHub is reconnected | Disclosure (21 Oct), reset guide (22 Oct) |
| PingCAP / TiDB Cloud | Active management API keys, replace then revoke | Business and billing details were in exported records, so expect targeted phishing | Customer actions (20 Oct) |
| Zeabur | All variables, access tokens, zSend keys, database passwords inside each database, n8n encryption keys | Treat database backup data in two named regions as leaked; rebuild affected dedicated servers | Report (19 Oct) |
| LiteLLM 1.107.0 | Host cloud role, every provider key in the proxy database, master and virtual keys, DATABASE_URL password, any secret in a process environment | Clean the host first; a rotated key typed on an infected host leaks again | Incident report (21 Oct) |
Railway's preliminary report recommends that every customer who signed up before 21 October rotate keys stored there, and says its final forensic report was still pending at our 24 October check. Zeabur dates its wind-down to 1 to 31 December (Zeabur notice), so anything left there needs a migration plan as well as a rotation.
What is the runbook, step by step?
Work in this order.
- List every secret. Platform variables, CI secrets,
.envfiles on servers, tokens in databases, and secrets someone typed into an SQL editor or a chat. Note who issued each. - Preserve evidence. Export audit logs from the platform, GitHub and cloud account before you revoke anything. The FTC tells businesses not to destroy forensic evidence while remediating, and Railway's GitHub guide says to export the log first.
- Clean the machine, if the machine ran malicious code. See the LiteLLM section below. Rotate from a clean system.
- Revoke at the issuer. The cloud console, the payment dashboard, the AI provider, GitHub. GitHub calls revoking with the provider the most important step. If the secret was committed to a repository, revoke first and clean history second.
- Change database passwords inside the database. Then update every consumer.
- Redeploy and restart. Anything that loaded the old value at start still holds it.
- Prove it. The old credential must fail against the provider. The new one must work. Check logs for authentication failures from services you forgot.
- Hunt what was left behind. See the next section.
- Write it down. Date, what was rotated, who did it, what evidence you kept.
What do I check for misuse?
Use the issuer, because the breached platform's own logs are the weakest record. Look at usage grouped by key and the issuer's audit log. For AWS, LiteLLM suggests searching CloudTrail for GetCallerIdentity calls from the host's role, then for use of its credentials from outside your infrastructure. For GitHub, Railway's guide points to the security log (90 days) and the organization audit log (180 days).
For PostgreSQL, Railway's guide suggests checking pg_roles for unexpected roles and pg_stat_activity for sessions you do not recognize after the password change. PostgreSQL does not log successful connections by default, so a quiet log proves little.
What can survive a rotation?
Anything the attacker added with a stolen secret. Railway's GitHub guide warns that an attacker with push access could add a webhook or secret that outlives a password reset. Check deploy keys, webhooks, repository secrets, collaborators and authorized OAuth apps. Zeabur tells owners of affected servers to look for SSH keys they did not add, unfamiliar privileged accounts and cron entries, and prefers a rebuild.
What if the breached thing ran code on my machine?
Clean first, then rotate. LiteLLM's report lists the order: confirm exposure, do not start Python or pip in the affected environment, remove the startup files, stop the background process, rebuild the environment pinned to an earlier release, and only then rotate. If the host had outbound internet while the bad version was installed, assume the secrets left. A new key created on a machine that is still running the stealer is a new leaked key.
What does rotation not fix?
Data already read. Zeabur's report says rotation does not invalidate data that was already read, so secrets, hashes and tokens stored in a database need replacing too. Session tokens, password hashes and API keys held in a table are in scope.
It does not answer the notification question either. If customer data sat in a store the attacker could read, the FTC's guide points to breach notification laws and law enforcement. That is a legal question for you or your counsel.
What do I ask the host or vendor?
Which of my stored secrets were readable in the incident, and in what form? Please tell me the date range of the exposure, and whether my account or workspace was among those you notified or ruled out.
Which credentials did you rotate on my behalf, on which dates, and where in my audit log do those entries appear? Which ones do I still have to replace myself?
If the answer to the first question is "we cannot say", rotate everything in scope. See who is responsible for patching and fixing for the wider split of duties between you and a host.
Am I affected? Where you stand right now
A lookup from outside can see keys left in your public pages or files. It cannot see what a hosting platform stored for you or whether a worker was restarted.
- Detected A vendor notice or a primary-source report names a platform or package you use, and you stored secrets there. Rotate everything in scope now, in the order above, without waiting for confirmation from the vendor.
- Unknown You use a platform that disclosed an incident, but have no notice, or cannot say which secrets it held. The common case. List what was stored, treat it as in scope, and ask the vendor the questions above.
- Confirmed by owner evidence The issuer's own log shows the old credential revoked, shows no use of it after the exposure window, and the new one in use from a process started afterwards. The rotation is done. Keep the evidence. Whether the old value was used before you revoked it is a separate question for the issuer's logs.
No scan, ours included, moves you from "unknown" to "confirmed".
What can a scan add?
Red's free scan looks from outside for API keys and tokens in shipped bundles and for exposed .git and .env files. That finds secrets in places the public can reach. It cannot see variables held inside a hosting platform, so it cannot tell you whether a rotation is complete.
Questions people ask
My host says only a subset of customers was affected. Do I still have to rotate?
If you stored secrets there, yes. Railway's own disclosure said the impact might be limited to a subset of users and still asked all users to roll every variable while it narrowed the scope. You cannot see from outside whether you are in the subset, so the cost of rotating is smaller than the cost of guessing.
I deleted the .env file from my repository. Is that enough?
No. GitHub says the most important step is revoking the secret with the provider that issued it. Removing it from the code, or even from git history, does not make a copy that someone already read stop working.
What should I rotate first?
Start with credentials that can create or read other credentials: cloud account keys, platform and CI tokens, GitHub tokens. Then database passwords, then paid API keys such as payment and AI providers, then signing and encryption keys. That order is our judgment; the vendor reports in this guide agree on the principle that the broadest key goes first.
Will rotating keys take my site down?
It can. A key revoked before the replacement is deployed stops working immediately. If you have no reason to think the old key was read, overlap the two for a short window. If you think it was read, revoke first and accept a short outage, then redeploy.
I changed the password variable in my hosting dashboard. Is the database password changed?
Not necessarily. Zeabur says changing the environment variable does not change the database password, and Railway says dependent services keep the old password until they are redeployed. Change the password where the database lives, then update and restart every service that uses it.
How do I know whether someone used my old keys?
Ask the issuing provider, not the host. Look at usage by key, the provider's audit log and the old key's last-used time. Absence of activity in a log you cannot fully trust, or one that was never switched on, does not prove the key was never used.
Sources
- Railway: Security Disclosure, Wednesday October 21st (21 October 2026)Environment variables may have been accessed and some decrypted. Asks all users to roll every variable and registry credential. Checked 24 October 2026.
- Railway: Preliminary Report on the October security incidentRecommends that customers who signed up before 21 October rotate keys stored on Railway. Preliminary; the final forensic report was pending at our check.
- Railway status: incident QH8L8B3EDated updates: builds paused, bucket credentials rotated, all API and project tokens invalidated 22 October 04:39 UTC.
- Railway: redeploy buckets and reconnect GitHub after the credential resetRevoke the old GitHub registry token at GitHub as well; redeploy consumers; check audit logs.
- Railway: rotate PostgreSQL database credentialsRegenerating redeploys the database only; dependent services keep the old password until redeployed.
- Railway: check for anomalous GitHub accessExport the audit log first; check deploy keys, webhooks, secrets and collaborators.
- PingCAP: TiDB Cloud security update, October 2026 (19 October 2026)Intrusion 9 September to 7 October using stolen service-account credentials. Our direct fetch returned HTTP 403 on 24 October; facts taken from the research snapshot of 22 October.
- PingCAP: customer actions, October 2026 (20 October 2026)Asks organization owners to replace active management API keys and to verify unexpected requests. Same 403 caveat as above.
- Zeabur: October 2026 security incident, preliminary report (19 October 2026)Rotation list, the database-password caveat, and the host rebuild advice. Preliminary; the report says it will be updated.
- Zeabur: Zeabur is shutting down (17 October 2026)Wind-down dates: servers expire from 1 December, final shutdown 31 December.
- LiteLLM: security incident of 21 October 2026Affected artifact windows, clean the host first, then rotate. Last updated 21 October 20:30 UTC at our check.
- GitHub Docs: remediating a leaked secretRevoke with the provider first; cleaning git history is a later step.
- GitHub Docs: token expiration and revocationA revoked token cannot be restored; revoking an OAuth authorization revokes its tokens.
- OWASP Secrets Management Cheat SheetIncident response order: revoke, rotate, then delete from exposed systems.
- Stripe Docs: API keys, rotate safelySeven-day overlap option when rotating in the Dashboard; choose Now to delete the old key.
- Google Cloud: best practices for managing service account keysDisable a key before deleting it; replace, then invalidate.
- FTC: Data Breach Response, a Guide for BusinessChange credentials, keep forensic evidence, review what service providers can access.
Facts on this page are dated and point at the primary source next to them. If a vendor status changes after the date shown, the vendor's page wins; tell us at support@run.cloud and we will update it.