Use lines picked as the denominator, record zone/shift/device/firmware and check whether report definitions changed.
EVIDENCE LAB / 02 / QUALITY OPERATIONS
The errors rise.But for whom?
A familiar workplace accusation meets denominator discipline and a controlled test. Inspect the data before trusting the loudest explanation.
FICTIONAL3 STAGESCOMPARE THE RATE
> open /lab-02/brief.txt
The warehouse scanning mystery
In a fictional outbound warehouse, wrong-item reports jump after a routine scanner firmware update. A supervisor suspects an evening shift's training. Another suggests shelf labels. A device analyst asks whether the scanner itself changed.
21/1kAFTER / MODEL X
3/1kBEFORE / MODEL X
4/1kMODEL Y / SAME PERIOD
Mission: decide which data deserve attention, design a fair test, and prevent a premature “people problem” conclusion. Rates are invented and measured as wrong-item reports per 1,000 picked order lines.
THE INVESTIGATION BRIEF_
Compare exposure before attributing blame
Does the increase track a shift, the device model, a firmware change or an interaction among them?
Useful lens: Pareto and exposure-normalized rates to focus the question; Fishbone to map alternatives; controlled comparison to test.
What the record establishes
- Model X's sample rises from 3 to 21 wrong-item reports per 1,000 lines.
- Model Y remains near 4 per 1,000 in the illustrative comparison period.
- Firmware X-4.8 preceded the increase and a controlled trial produces divergent rates.
What remains open
- Whether the result generalizes to all warehouse zones and order mixes.
- Whether the recorded wrong-item reports are equally complete across shifts.
- Whether the newer firmware creates a throughput or accessibility benefit that a rollback could remove.
Evidence boundary
Three samples and a small trial are instructional, not a statistical power analysis. Matching conditions reduces confounding but does not eliminate it. Operational context and data quality still matter.
STAGE 01 / DEFINE THE COMPARISON
“The evening shift is careless.” Is that evidence?
A daily report shows the largest number of wrong-item errors comes from the evening shift in Zone C. But Zone C also processes the largest number of lines, and several shifts use Model X scanners.
DECISION / WHAT SHOULD YOU COMPARE FIRST?
A — Normalize by orders and break down by scanner model
Appropriate. Use errors per 1,000 picked lines rather than total count, stratify by shift, model and zone, and compare the same exposure before and after the update. The apparent shift effect may be explained by equipment allocation.
Continue to evidence →B — Require retraining for the evening shift based on raw totals
Not supported. High counts may reflect high volume. Retraining is not necessarily harmful, but it can waste effort, stigmatize staff and delay the investigation. First compare rates and exposures before asserting a cause.
Continue to evidence →STAGE 02 / INSPECT THE PACKETS
What changed, and for whom?
EVIDENCE A / Exposure-normalized sample
Three illustrative samples each cover 5,000 picked lines.
| Group | Wrong-item reports | Rate |
|---|---|---|
| Model X / prior week | 15 | 3 per 1,000 |
| Model X / after update | 105 | 21 per 1,000 |
| Model Y / same new week | 20 | 4 per 1,000 |
Order mix, staffing and zone allocation are possible confounders. These counts identify an association, not the full cause.
EVIDENCE B / Device trace and change log
Firmware X-4.8 was deployed to Model X before the first increase. Model Y did not change. In a sample of 50 Model X wrong-item reports, 37 show a delayed server acknowledgement and a repeated scan event. The device log cannot alone establish how the second event led to the wrong item.
EVIDENCE C / Controlled device comparison
During an approved representative trial, eight Model X devices are returned to their previously validated firmware while eight matched devices remain on X-4.8. Staff, order mix and sampling windows are kept comparable. The trial group records approximately 4 errors per 1,000 lines; the updated-firmware group remains near 20 per 1,000.
The result supports the firmware-path explanation, but a broader rollout and observation period are still needed to establish durable improvement.
DECISION / WHICH INVESTIGATION ACTION MAKES THE CLAIM STRONGER?
A — Use a controlled firmware comparison before blaming the operator
Stronger evidence. Hold staff and order mix as comparable as practical, compare updated and prior firmware on the same device class, and inspect the acknowledgement/repeat-event path. Include a rollback plan and a clear outcome threshold.
Continue to the control decision →B — Rearrange every shelf immediately because errors involve wrong items
Premature. Shelf layout may be a contributor in some incidents, but the sharp device-model distinction and firmware change deserve a discriminating test before a disruptive global redesign.
Continue to the control decision →STAGE 03 / PREVENT RECURRENCE
The trial helps. What counts as completion?
A controlled rollback on a small set of devices substantially reduces errors. Operations wants to mark the problem resolved immediately, yet the full fleet and all shifts have not been observed.
DECISION / WHAT IS A DEFENSIBLE NEXT STEP?
A — Stage a verified remediation and monitor error rate by model
Defensible. Implement the approved rollback or corrected firmware in stages. Preserve successful scan validation, track wrong-item reports per 1,000 lines by model and shift, and watch for an increase in scan latency or unconfirmed orders. Verify across representative busy periods before closing.
Continue to the debrief →B — Close the investigation after one low-error hour
Insufficient observation. A short interval is vulnerable to load, shift and order-mix changes. The measure must include a representative recurrence cycle and an exposure denominator.
Continue to the debrief →> lab-02 / reasoning-review
Debrief: counts were not the cause.
- Observed: Model X errors increase from 3 to 21 per 1,000 lines; Model Y remains near 4 per 1,000; a Model X firmware update preceded the change.
- Working hypothesis: a new scan-acknowledgment behavior in firmware
X-4.8contributes to repeated or misattributed scans. - Discriminating evidence: a controlled comparison with previous validated firmware produces materially lower errors under comparable operations.
- Safe control: an authorized staged remediation with device validation, operating procedures and rollback readiness.
- Verification: multi-shift error rate per 1,000 lines, stratified by device, alongside scan latency, throughput and unintended side effects.
Reasoning workshop
Read the denominator carefully
In the invented sample, 105 errors in 5,000 lines equal 21 per 1,000. The previous Model X sample records 15 in 5,000, or 3 per 1,000. The observed rate ratio is seven, but that number alone does not isolate the firmware from differences in workload or reporting.
What might disconfirm the firmware hypothesis?
If error rates remain high on devices with the earlier validated firmware under matched conditions—or if a different scanning pathway shows the same increase—revisit layout, order substitutions, upstream catalog data and process timing. Avoid labelling operators as the cause without supporting evidence.
Verification plan
Keep order mix and staffing comparable; capture scan acknowledgement, error rate, throughput and staff feedback.
Track multiple shifts and workload peaks, validate the corrected firmware path, and watch for new latency or missed-scan effects.
Takeaway: A large count can tell you where work is concentrated; a fair rate and a discriminating test help explain what changed.
TRANSFER THE METHOD_
Pareto and defect rates can tell you where to look. A controlled comparison helps establish what changed. Neither a loud complaint nor the largest raw count proves who or what caused the outcome.
Continue the Evidence Lab
LAB 03 / Migration →Test a decision that starts with hard requirements, not preference scores.
All equipment names, measurements and outcomes are fictional. Operational modifications require authorized review and change controls.