The 14:00 payment failure
Recurring incidents, a discriminating test and an imperfect workaround.
Read payment case →Academy / Chapter 03 / Applied investigations
Four fictional professional scenarios. Every number, system, company and outcome below is invented for training—not a claim about real organizations or verified incidents.
NEW / GUIDED INVESTIGATIONS_
The professional Case Files show worked conclusions. The Evidence Lab offers three staged fictional investigations where you inspect evidence, choose a reasoning path and reveal feedback without submitting anything.
CASE DIRECTORY_
Recurring incidents, a discriminating test and an imperfect workaround.
Read payment case →Prioritize the recurring burden without assuming a Pareto bar is the cause.
Read warehouse case →Compare what is and is not affected before choosing a repair.
Read facilities case →Separate required conditions, preferences and downside risk.
Read migration case →CASE P01 / 5 WHYS + EVIDENCE TESTS
Initial deviation. A fictional payment service reaches 8.0% failed authorizations between 14:00 and 14:08 on four weekdays, against a 0.2% normal failure rate. Other time windows remain normal.
| Signal | Classification | Reason |
|---|---|---|
| Failed requests and database lock waits rise in the same interval. | Observed | Timestamped metrics and logs. |
| The export/report job starts at 14:00 on affected days. | Observed | Scheduler event records. |
| The report job causes the payment failures. | Hypothesis | Timing alone does not establish causation. |
| A missing concurrency limit allowed the report to occupy payment resources. | Working explanation | Requires a controlled test and code/configuration review. |
Under a controlled test, postponing the report while leaving payment demand comparable coincides with a return to baseline, and reproducing the report under staging-like load recreates similar contention. This strengthens the hypothesis, although monitoring should continue for alternative causes.
Workaround: postpone the scheduled report with clear ownership and a business-impact warning. Durable control: isolate workload capacity or enforce concurrency and lock-duration controls. Verification: compare 14 comparable peak windows before and after, reviewing failure rate, job completion and lock duration.
Restoring transactions is incident recovery. Establishing why the job caused contention and designing an independently verified control is problem management.
CASE P02 / PARETO → FISHBONE
Initial deviation. Over one fictional month, an outbound warehouse records 200 defect reports. The team wants to “fix everything” but has only one improvement cycle available.
| Defect category | Reports | Share | Cumulative |
|---|---|---|---|
| Wrong item picked | 92 | 46% | 46% |
| Missing item | 48 | 24% | 70% |
| Label mismatch | 30 | 15% | 85% |
| Package damage | 20 | 10% | 95% |
| Other | 10 | 5% | 100% |
The first three categories account for 85% of recorded events. This is a useful starting priority for further analysis. But these are counts, not costs, safety severity or validated causes. Different exposures or under-reporting could change the ranking.
The team samples 30 wrong-item reports, compares shifts and zones, and maps candidate causes with Fishbone: ambiguous shelf layout, scanner latency, training gaps, substitute item labelling and workload peaks. A test on one affected zone supports shelf placement as one contributor, but it does not rule out the other branches.
Control: trial a clearer placement and independent scan confirmation in one zone. Verification: compare errors per 1,000 lines picked—not just raw counts—and include labor time and new failure modes.
The largest category is the best place to investigate next, not necessarily the easiest or best place to invest. Compare severity and controllability before deciding.
CASE P03 / KEPNER–TREGOE PROBLEM ANALYSIS
Initial deviation. A fictional office building's north-zone meeting rooms are 3–4°C warmer than set point after 13:00. South-zone rooms are normal, and the north zone was normal until a recent maintenance visit.
| Dimension | IS | IS NOT |
|---|---|---|
| What | North-zone supply temperature above expected | All building zones failing |
| Where | North rooms on one air-handler branch | South-zone rooms |
| When | After 13:00 since maintenance | Before midday and before maintenance |
| Extent | 3–4°C excess during afternoon load | Total loss of cooling |
Review differences between the two branches and changes made during maintenance. One plausible cause is a reversed actuator control signal installed in the north-zone valve. But being the newest change is not proof.
A qualified technician verifies command versus physical valve position under controlled safe conditions, finding reverse movement on the affected branch while the comparison branch responds correctly. Workaround: controlled set-point and occupancy adjustment within safe limits. Correction: restore the correct control direction under change control.
Verification: compare time-at-set-point over a representative week and monitor for unintended humidity and energy consequences.
IS/IS NOT narrows the test. Do not repair solely because a recent change appears suspicious, especially in systems requiring licensed or qualified personnel.
CASE P04 / KEPNER–TREGOE DECISION ANALYSIS
Decision to make. A fictional team must move a reporting workload. The two technically eligible options are a managed service and a self-hosted platform. Another option fails mandatory residency requirements and is rejected before scoring.
MUST: meet the data residency requirement, meet the recovery objective and have an independently supported export route. The options that fail a MUST do not earn eligibility through high preference scores.
| Weighted WANT | Weight | Managed (1–5) | Self-hosted (1–5) |
|---|---|---|---|
| Operational simplicity | 5 | 5 | 2 |
| Cost predictability | 4 | 3 | 4 |
| Customization | 3 | 2 | 5 |
| Recovery operations | 4 | 4 | 3 |
| Illustrative weighted total | 16 | 59 | 53 |
The managed option scores 59 against 53 using these invented weights and scores. Those values are judgments, not measured probabilities. A sensitivity test asks whether changing the weight of customization, or discovering a new vendor dependency, would reverse the preference.
Decision condition: proceed only if the vendor's exit test and recovery drill meet the MUSTs. Potential problem: higher switching cost if the export path fails in practice. Protective action: conduct a pilot extraction and rollback rehearsal before commitment.
Decision scoring makes trade-offs visible. It cannot make preferences objective, guarantee success or replace checking downside risks.
TRANSFER YOUR LEARNING_
Use eight reasoning challenges with worked explanations.