Keep token-signature validation active, confirm the permitted refresh endpoint, record the configuration and protect the event timeline.
EVIDENCE LAB / 01 / SERVICE RELIABILITY
One region fails.Why not both?
Start with a disputed outage, review logs and changes, test the competing stories and choose a safe next step. Explore alternate responses and uncover where the evidence leads.
FICTIONAL3 STAGESTEST THE THEORY
> open /lab-01/brief.txt
The regional sign-in outage
A fictional organization uses two independently deployed identity-gateway regions. After an identity-provider signing-key rotation, login failures spike in the West region. The on-call team is under pressure to declare that the external identity provider is down.
26%WEST LOGIN FAILURES
0.4%EAST LOGIN FAILURES
09:00KEY ROTATION UTC
Mission: restore trustworthy authentication, identify why West differs from East, and select a fix whose effectiveness can be verified. All data and outcomes are fictional.
THE INVESTIGATION BRIEF_
Separate regional symptoms from a shared cause
What explains West failing when East continues to verify the same class of token?
Useful lens: Kepner–Tregoe-style IS / IS NOT comparison, followed by a controlled contrast.
What the record establishes
- West's sampled errors begin after the key rotation and include an unknown key identifier.
- East verifies new-key tokens in the same interval.
- West has recent egress changes and refresh timeouts.
What remains open
- Whether the egress revision is the complete cause or one contributing control failure.
- Whether the canary result generalizes across the West fleet and peak demand.
- Which change validation would have detected this before the rotation.
Evidence boundary
The fictional log extracts show associations and a canary test, not a randomized production experiment. A successful canary narrows the explanation; it does not prove every gateway is healthy.
STAGE 01 / START WITH THE DEVIATION
Two regions. One alleged outage.
At 09:08 UTC, successful login traffic falls in West, but East remains within its usual error range. The identity provider reports normal availability. A recent signing-key rotation is visible in the change calendar, but its role is not yet established.
What is your first move? Choose a response to reveal its consequence. Think about what each response would reveal.
A — Compare the affected and unaffected regions
Promising triage. You preserve the distinction between West (IS failing) and East (IS NOT failing). Compare error type, token metadata, recent network changes and key-refresh results. The rotation may be a trigger, but the regional difference needs explaining.
Continue to the evidence review →B — Restart every identity gateway and declare a provider outage
Weak and risky. A restart might briefly reset state, but it destroys useful timing evidence and does not explain why East still works. Calling the provider unavailable contradicts available comparative evidence. Keep service restoration coordinated with the incident lead, then examine the regional difference.
Continue to the evidence review →STAGE 02 / INSPECT THE RECORD
Three evidence packets.
Read the packets in any order. Distinguish a correlation from an explanation and note which observation would make a proposed explanation less likely.
EVIDENCE A / Gateway verification logs
West 09:08–09:12 UTC: rejected tokens specify fictional key ID kid=2026-b; West's locally cached key set contains only 2026-a. Verification fails with unknown signing key.
East, same interval: its cache includes 2026-b and token verification succeeds. No evidence of a general identity-provider outage appears in the sampled East requests.
EVIDENCE B / Network and change history
A West-only egress allow-list revision was deployed at 07:50 UTC. Between 08:00 and 09:12, West's scheduled key-set refresh attempts to the approved provider endpoint time out. East's attempts succeed.
The identity provider rotated to key 2026-b at 09:00 UTC. The earlier egress change may explain why West could not refresh, but both the routing and cache behavior need testing.
EVIDENCE C / Controlled comparison
During an approved, isolated test, the team allows one West canary to reach the key-set endpoint and fetch the current keys. Its verification errors return to the normal range while comparable West gateways without the change still reject new-key tokens.
This supports a network-access and cache-refresh explanation; it does not establish that the entire incident has been permanently resolved.
DECISION / WHAT BEST EXPLAINS THE DISTINCTION?
A — Test whether the West egress rule blocked key refresh
Evidence-supported. The causal account now has discriminating support: the provider rotated signing keys, the West-only network revision prevented refresh, and restoring access on a canary allowed verification of new-key tokens. Check the rule's intended security boundary before expanding the change.
Continue to the control decision →B — Conclude that the signing-key rotation itself was defective
Overconfident. The rotation affected both regions; only West failed. The East comparison and West-only egress timeouts point toward refresh access, not necessarily a faulty provider key. Investigate the regional difference before assigning responsibility.
Continue to the control decision →STAGE 03 / RECOVER WITHOUT WEAKENING TRUST
What should the team change?
The canary test supports the hypothesis. Production recovery still requires approved access boundaries, monitoring and a rollback plan.
A — Repair approved key-refresh access and verify across cycles
Defensible action. Restore the narrowly authorized network route under change control, monitor successful key-set refreshes and authentication error rates, and check a future rotation or simulated refresh before closing. Preserve validation of token signatures at all times.
Continue to the debrief →B — Disable token verification so all logins succeed immediately
Unacceptable control. Removing authentication validation creates a serious security exposure and masks the actual fault. Incident urgency does not make a control bypass a reliable fix. Use a properly authorized remediation or safe service-degradation process.
Continue to the debrief →> lab-01 / reasoning-review
Debrief: the regional difference mattered.
- Observed: 26% West errors versus 0.4% East; West rejected new-key tokens; West key refreshes timed out after an egress change.
- Hypothesis: a West-only egress rule prevented the gateway from retrieving the identity provider's current signing keys.
- Discriminating test: a canary with approved access fetched the key set and verified new-key tokens while untreated peers did not.
- Control: authorize and monitor narrow key-refresh access, verify signature checks remain intact, and review change validation around credential rotations.
- Verification: track normal login error rates in both regions over comparable load and at least one controlled refresh or key-rotation exercise. An initial success does not justify immediate closure.
Reasoning workshop
How to construct the causal claim
The proposed explanation is a chain: West-only access change → failed signing-key refresh → stale key cache → rejection of newly signed tokens. Every arrow needs support. The time sequence motivates a test, while the canary that refreshes and succeeds provides evidence for the mechanism.
What could falsify it?
If West still rejected kid=2026-b after a confirmed successful key refresh, the access failure would be insufficient to explain the remaining symptoms. Investigate token validation, clock alignment, issuer metadata and other distinct causes rather than forcing the original theory.
Verification plan
Monitor refresh success, gateway error categories and regional login rates under comparable load; retain a defined rollback trigger.
Observe at least one planned key refresh or controlled rotation, compare West against East and document the reason the egress rule escaped change review.
Takeaway: For an apparently global dependency outage, compare a working population before blaming the shared provider.
TRANSFER THE METHOD_
Use IS/IS NOT comparisons to make hypotheses specific. The first change before an incident may be relevant, but you still need a test connecting it to the observed failure.
Continue the Evidence Lab
LAB 02 / Warehouse →Different domain, same discipline: compare the rate of defects rather than blaming a shift.
Fictional case for critical thinking. Not deployment, security, or emergency-response instructions. Production system changes require authorized personnel and relevant controls.