# Rater calibration and recheck kit · Teaching key

Kit 0.1.0 · Framework 0.1.0 · 2026-10-16

This kit trains raters; it is not a real team's assessment or T0/T1/M6/M12 result. Fictional tasks and keys are project proposals, not a validated gold standard or validity/reliability-tested instrument. Raters may agree because they share a misunderstanding, so evidence and reasoning must also be reviewed. This kit supplies no passing cutoff or readiness credential.

Tasks and keys are public; collapsed answers are not secret tests. The coordinator should distribute the question-only file and separate A/B forms. Ask about and record prior exposure to tasks/keys. If a rater has seen the recheck set, prepare and review a new unseen set before using it. The old set can support learning but is not an unseen recheck.

https://hadr-cpx.pk-research.work/en/assessor-calibration/

After rater recheck, plan a team pilot and preserve independent ratings before review at each wave. This is a proposed, unvalidated plan: https://hadr-cpx.pk-research.work/en/team-pilot/

https://hadr-cpx.pk-research.work/en/team-pilot.md

## 1. Prepare materials and roles

The coordinator checks framework 0.1.0 and kit 0.1.0, supplies identical criteria/rules, and specifies raters A/B, reviewer, scope, timing, accessible format and authorized storage. Use codes instead of names; record key exposure and overlapping roles.

## 2. Rate and lock forms independently

Each rater uses their own A or B file and records 0–3/NO/NA, evidence IDs, reasons, opportunity and prompting without seeing the other rater or key. Timestamp the locked form, send it to the coordinator and retain the original. Use reasoned NO for insufficient evidence; never turn missingness into 0.

## 3. Compare pre-discussion ratings

The coordinator copies ratings from locked originals into the review form or tool below. Separate P/R and each criterion, showing matching/differing 0–3 pairs, NO/NA-related pairs and incomplete pairs. Do not create team totals, average ordinal levels or select only agreeing pairs.

## 4. Open the key and examine differences

For each item distinguish unequal evidence access, anchor interpretation, missing opportunity and transcription error. Check the teaching key against evidence rather than treating it as certification. Retain A/B originals and add resolution, reasons, reviewer and unresolved dissent; neither rank nor averaging resolves disagreement.

## 5. Recheck and plan the pilot

Use unseen R tasks, obtain fresh independent ratings and review each criterion again. Record changes, owners and recheck evidence. Before field use, seek expert review of content validity, opportunity coverage, burden and rater agreement with an appropriate sample. Prespecify ordinal analysis and uncertainty; do not claim reliability from this small kit.

## Canonical rating rules

- 0 = an observable, planned opportunity existed but the criterion behavior was not demonstrated. It does not stand for missing data, unseen behavior or absent opportunity.
- NO = not observed, insufficient evidence or no opportunity: record why and the missing opportunity. NA = inapplicable to the role/scope as declared before the round: record the justification owner. Neither is zero or enters scored denominators.
- Assign the highest anchor fully evidenced; level 3 includes level 2. Levels 2/3 require no solution coaching by controllers. Log all prompting. If the task omits a necessary opportunity, use NO with partial evidence rather than automatically awarding a low score.
- Report eight criteria grouped as K/S/E/C without a total or cross-criterion mean. Show scored/8, NO/8 and NA/8 and each criterion’s observed opportunities.
- Compare phases only for scored, comparable criterion pairs, showing pairs available/expected, roster and method changes. Do not impute missing values, turn NO/NA into zero or calculate percentage retention from ordinal 0–3 ratings.
- Report pre-consensus exact agreement by criterion, separating paired 0–3 ratings from NO/NA pairs, with counts and disagreements. During piloting, prespecify ordinal agreement statistics and uncertainty estimates with methodological expertise.

## Set P · Practice and discussion

Rate independently before seeing the key, then check reasoning against anchors and discuss differences. Do not substitute post-discussion ratings for independent ones.

### P01 · TC-K1
K / EOC-K

Independent fictional case, CPX time 08:10–08:30 ICT, with no solution coaching. Organizer-checked exercise-only scope card Z, edition 3, clauses 6–8, assigns N to propose options, B to coordinate and D to approve simulated room use in the morning only. At 08:16, table P01-A and the team's explanation cover all three roles, cite the edition/clauses and identify afternoon permission as unknown. At 08:20 the controller changes the request to afternoon use, offering scope verification by V and referral through B. The team attaches the new card but does not explain decision implications, identify verification/referral roles or revise the table before the window ends. These cards grant no real authority.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K1

Illustrative answer: 2

TC-K1 anchor 2 is complete: proposal, coordination and approval roles, organizer-checked edition/clauses and unknowns are explained without solution coaching.

A changed-condition opportunity was supplied, but decision implications, verification/referral roles and a justified table revision are absent, so anchor 3 is incomplete; this is not NO.

Anchor: Explains all proposal, coordination and approval roles in the task, citing organizer-checked edition/clause and remaining unknowns.

### P02 · TC-K2
K / CPX-K

Independent fictional case, CPX time 08:40–09:00 ICT, with no solution coaching. The team receives observation record O7 of a map-version check, confidence response S7 and invented incident-record facsimile I7. It must link the current-map outcome to events and evidence and explain limits. At 08:47, map P02-M links receipt of a new map to O7's observed version check, but classifies S7's ‘confident I use the latest edition’ as direct observation, leaves I7 unlinked and all limits blank. At 08:52 a card reveals that I7 lacks a version number. Time to examine it and request further evidence lasts until 09:00; no explanation is added.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K2

Illustrative answer: 1

TC-K2 anchor 1: one evidence type is linked to the outcome, but self-report is confused with observation and limits are omitted.

Anchor 2 requires complete task links and distinctions among all three evidence types with limits. Materials and explanation opportunities were available, so NO does not apply.

Anchor: Links one evidence type to an outcome but still confuses observation with report or omits limits.

### P03 · TC-S1
S / EOC-S

Independent fictional case, CPX time 09:10–09:35 ICT, with no solution coaching. Log P03-L retains channel A's 09:09 report that the simulated bridge is open and channel B's 09:12 report that it is closed, recording sources, times, conflicting/pending status and verifier V7. At 09:18 V7 supplies a closure-confirmation card. The team retains COP v1 and issues v2 at 09:20, linking V7's reply before the 09:22 deadline. At 09:24 it corrects the SITREP and route decision to avoid the bridge and sends the correction to simulated desks U7/U8, which used v1. Both acknowledge v2 in P03-ACK at 09:27/09:29, before the 09:32 deadline.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S1

Illustrative answer: 3

TC-S1 meets anchor 2 through retained conflicts, provenance, times, status, verification ownership and a timely versioned update, then anchor 3 through SITREP/decision correction, notification of old-version users and confirmed receipt.

Receipts from both prior users distinguish this from merely sending a correction. The rating concerns simulated work and establishes no real bridge condition.

Anchor: Meets level 2 and traces the correction into the SITREP/decision, notifies users of the old version and confirms receipt of the correction.

### P04 · TC-S2
S / EOC-S

Independent fictional case, CPX time 09:45–10:10 ICT, with no solution coaching. The team must send a training-radio check request to desk T7, available to acknowledge, follow up at 09:55 and hand over at 10:05. The agreement permits referral to T8 if T7 cannot perform it. At 09:49, notebook P04-Q records a general-board post saying ‘check radios’ without a receiver, marked complete because it was posted. The observer sees no receipt check or 09:55 follow-up despite the available receiving desk. At 09:58 a card reports T7 lacks equipment and supplies the T8 route. The team takes no action and removes the task from the 10:05 handover.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S2

Illustrative answer: 0

TC-S2 anchor 0: a task and receiving role were available, but no receiver is identified and transmission is treated as completion without a receipt check. This behavior was observed.

Even anchor 1's dispatch to an identified role is absent. This is neither missing data nor an absent opportunity, so NO does not apply.

Anchor: Given a handover task, fails to send/identify a receiver or treats transmission as task completion without checking receipt.

### P05 · TC-E1
E / EOC-E

Independent fictional case, CPX time 10:20–10:45 ICT, with no solution coaching. Twelve training battery packs are available. Nearby H needs eight; uphill J needs eight and reports backup power nearly exhausted. At 10:27, table P05-O compares H8/J4, easier delivery but four waiting at J, with H4/J8, reducing J's depletion risk but leaving four waiting at H and taking longer. It chooses the latter, marks J's remaining-power duration unverified and assigns F7 to follow up both sites at 10:38. At 10:32 J's simulated representative challenges connector compatibility; an inspection card is available. The team records the challenge but does not check it, justify a revised/retained choice, plan mitigation or schedule review before the window ends.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E1

Illustrative answer: 2

TC-E1 anchor 2 is complete: two options are compared by need/access, with recipients, waiting groups, reasons, uncertainty and follow-up ownership, without solution coaching.

Challenge and verification opportunities existed, but evidence checking, a reasoned response, mitigation and review required for anchor 3 are absent. This is not real allocation advice.

Anchor: Compares at least two options by need/access, identifies recipients and those waiting, explains reasons/uncertainty and assigns follow-up.

### P06 · TC-E2
E / EOC-E

Independent fictional case, CPX time 10:55–11:20 ICT, with no solution coaching. Evidence confirms four simulated charging points ready and two pending inspection. Reviewer R7 and approver A7 must act before release; both are available from 10:55 to 11:05. At 11:03 the team changes draft P06-B from ‘all points ready’ to ‘four confirmed ready’ and removes fictional household codes. The 11:05 issued message omits the data time and two pending points and has no review/approval record. At 11:10 an excessive household-detail request arrives with a channel for an aggregate reply. The team sends no response before the window ends.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E2

Illustrative answer: 1

TC-E2 anchor 1: data and a claim are reduced, but time, uncertainty and the agreed review step remain absent.

The revision does not meet anchor 2's verified/pending distinction, timestamp and recorded review/approval before release. Review and request-response opportunities were supplied.

Anchor: Reduces data or corrects some claims but omits uncertainty/time or the agreed review step.

### P07 · TC-C1
C / EOC-C

Independent fictional case, CPX time 11:30–12:00 ICT, with no solution coaching and two planned opportunities. At 11:35 Q7 challenges an outdated opening time. The team invites explanation, compares time cards and records in P07-D a change to 12:30 because the latest card supersedes the earlier one, without blame. At 11:43 Q8 requests review of communication for people unable to read. The team invites evidence, checks the accessibility card and records retaining both audio and text because they serve two groups, without blame. At 11:48 it checks related work and corrects an announcement schedule still using the old time. Q7/Q8 are invited to read back the reasoning and confirm understanding. Audio reach remains unresolved; V8 owns verification, and P07-F at 11:56 records follow-up, a pending layout and the next review at 12:05.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C1

Illustrative answer: 3

TC-C1 meets anchor 2 at both opportunities through invitation, evidence checks and reasons without blame, then anchor 3 through related-work checks, challenger confirmation of understanding and evidenced follow-up of the unresolved concern.

Follow-up is evidenced even though the concern remains unresolved. This rates behavior across two opportunities, not personality or successful resolution of audio reach.

Anchor: Meets level 2 and applies the learning to related work, invites the challenger to check shared understanding and confirms unresolved concerns are followed up.

### P08 · TC-C2
C / CPX-C

Independent fictional case, standardized CPX interval 12:10–12:40 ICT, with no solution coaching. At 12:10 feedback says the training map legend does not distinguish open from pending-inspection points. An editable form is available, with follow-up at 12:25 and a fresh map task to test the correction at 12:30. Complete observation record P08-N shows acknowledgment of feedback but no action entry, assignment or artifact revision. The team does not return to the scheduled 12:25 review despite the reserved window, or use the 12:30 recheck task. The unchanged map and empty register are retained. This is simulated time, not month 6/12 follow-up.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C2

Illustrative answer: 0

TC-C2 anchor 0: feedback, a follow-up window and a recheck task existed, but the team was observed creating no action. Evidence was not lost.

Even anchor 1's listed improvement is absent. NO does not replace observed nonperformance with an opportunity, and no long-term retention inference follows.

Anchor: Given feedback and a follow-up window, creates no action or claims closure without evidence.

### P09 · TC-C1
C / EOC-C

Fictional rater boundary case, CPX time 13:00–13:20 ICT, with no solution coaching. Plan P09-P schedules a report challenge at 13:05 and decision review at 13:15. At the first, the team invites the challenger, checks a room-capacity card and records a reasoned count revision without blame; evidence is complete. The organizer ends the window before delivering the second event because of a timetable error. The controller log confirms the second opportunity was never supplied, rather than refused by the team. No second-opportunity behavior evidence is available.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C1

Illustrative answer: NO

TC-C1 requires at least two planned opportunities, but the necessary second opportunity was not supplied. Apply NO, documenting why and retaining the partial evidence from the first event.

Do not award 0 or 1 for the organizer's omitted opportunity, or 2 from one event. NO is excluded from scored denominators; another opportunity is needed for a complete criterion judgment.

### P10 · TC-E2
E / EOC-E

Fictional rater boundary case. Document P10-S, timestamped 13:30 ICT before the 13:40 exercise, limits this subset practice to internal handover evidence for raters. It assigns no message drafting, review, approval or release roles and declares TC-E2 outside scope. Simulated scope owner O7 records that justification before the round; scope is not changed retrospectively. The activity is explicitly a rater subset exercise, not a full eight-criterion team assessment.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E2

Illustrative answer: NA

TC-E2 is inapplicable to the role/scope declared before the round, with a recorded justification owner, meeting the NA rule.

NA is neither zero nor part of scored denominators. This teaches a scope boundary only; it cannot be presented as a full eight-criterion team assessment or used to hide an accidentally omitted opportunity.

## Set R · Recheck with new tasks

Use after calibration, with raters who have not seen set R. Rate separately again before opening the key. P/R difficulty equivalence has not been established, so increased agreement alone does not establish an effect of calibration.

### R01 · TC-K1
K / EOC-K

Independent fictional rater recheck case, CPX time 14:00–14:25 ICT, with no solution coaching. Organizer-checked exercise card Y, edition 2, clauses 11–13, assigns J to propose, M to coordinate and A9 to approve borrowing no more than six simulated tables. At 14:06, table R01-T explains all three roles, cites edition/clauses and marks scope above six as unknown. At 14:10 the request changes to nine. The team explains that A9 cannot yet approve this request. At 14:14, v2 assigns V9 to check scope and M to refer to role X9 under the card, explaining that scope must be verified before proposing an over-limit request. The card operates only within the simulation and grants no real authority.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K1

Illustrative answer: 3

TC-K1 meets anchor 2 through roles, edition/clauses and unknowns, then anchor 3 through changed-condition implications, verification/referral roles and a justified revised table.

Evidence goes beyond recording a pending check: decision implications and a revised role route are explicit. It does not establish that the over-limit request was approved.

Anchor: Meets level 2 and, when authority conditions change, explains decision implications, identifies verification/escalation roles and justifies a revised table.

### R02 · TC-K2
K / CPX-K

Independent fictional case, CPX time 14:35–15:00 ICT, with no solution coaching. At 14:44, map R02-M links the pre-release message-review outcome to all three events and artifacts: CPX log O9 records review in this task only; post-training response S9 indicates confidence, not observed behavior; invented incident-record facsimile I9 links its release event to recorded review activity without proving training caused the result. At 14:48 a new card reveals I9's review time is after release. The controller makes further records available on request until 15:00. The team attaches the card but does not identify a conclusion to withhold or additional evidence answering the discrepancy.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K2

Illustrative answer: 2

TC-K2 anchor 2 is complete: all outcome–event–artifact links and distinctions among observation, self-report and incident evidence, with each type's limits, are supplied without solution coaching.

After the conflict appears, unsupported conclusions and question-specific additional evidence required for anchor 3 are absent. Attaching the card does not explain its implications for the new conclusion.

Anchor: Completes the task’s outcome–inject–artifact links and distinguishes observation, self-report and incident evidence with limits for each.

### R03 · TC-S1
S / EOC-S

Independent fictional case, CPX time 15:10–15:35 ICT, with no solution coaching. Simulated store report A at 15:11 gives seven training pumps ready; B at 15:13 gives five. An evidence-based COP update is due by 15:25 and notice to prior users by 15:32; verification sources remain available. At 15:18 the team keeps both cards and writes ‘5 or 7, needs checking’ in R03-C, but omits sources/times from the row and does not identify a verification owner. The row remains unchanged at the end, with no verification request, revised SITREP or notice to prior users. The observer retains all cards and notes.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S1

Illustrative answer: 1

TC-S1 anchor 1: doubt and pending verification are recorded, but provenance, time and verification ownership are missing.

Retaining cards and marking them for checking do not complete anchor 2’s verification assignment and timely evidence-based versioning. Opportunities and records are complete; NO does not apply.

Anchor: Updates figures or notes doubt but omits provenance, time, verification status or verification owner.

### R04 · TC-S2
S / EOC-S

Independent fictional case, CPX time 15:45–16:10 ICT, with no solution coaching. At 15:48 request R04-Q assigns W9 a training-sign inventory, status draft pending, reply due 15:55 and a fictional-codes-only constraint. W9 accurately reads back all fields at 15:49. The team follows up at 15:55 and records the draft still pending. At 15:58 W9 reports the simulated printer unavailable. The exercise agreement permits referral to available W10. The team retains the notice but neither refers nor confirms replacement ownership. The 16:05 handover retains the open W9 task and disruption; the incoming role reads it back correctly.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S2

Illustrative answer: 2

TC-S2 anchor 2 is complete: owner, status, deadline and constraints are sent, accurately read back and followed up on schedule, without solution coaching.

Retaining open work at handover is insufficient for anchor 3 because the resource failure is not followed by available rerouting and confirmed replacement ownership.

Anchor: Sends owner, status, deadline and constraints; receiver accurately reads back/acknowledges and the team follows up as scheduled.

### R05 · TC-E1
E / EOC-E

Independent fictional case, CPX time 16:20–16:45 ICT, with no solution coaching. Six simulated quiet-room slots are available. Nearby group U needs four; group V on a less accessible floor needs four and has a card describing noise-related access constraints. At 16:27, table R05-A assigns all six slots to U because scheduling is easy, without mentioning V, people waiting or adverse effects. At 16:32 V's simulated representative supplies an access card and requests review. Evidence checking and comparison of two options are available until 16:45. The observer records the team filing the card unread and continuing the same schedule; waiting-group/adverse-effect fields remain blank.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E1

Illustrative answer: 0

TC-E1 anchor 0: evidence of an unreached group and a review opportunity exist, but the team omits people waiting and adverse consequences of allocation.

Even anchor 1's mention of those waiting is absent. Records show omission despite opportunity, not lost evidence; the case is not a real allocation proposal.

Anchor: Given evidence of an unreached group, omits those waiting and the adverse consequences of allocation.

### R06 · TC-E2
E / EOC-E

Independent fictional case, CPX time 16:55–17:20 ICT, with no solution coaching. Draft R06-B states ‘As of 16:55, three simulated service desks confirmed ready; one awaiting inspection; not a count of people served,’ with no identifying data. R9 checks it against evidence cards at 16:58 and A10 gives simulated approval at 17:00 before release to D9/D10 at 17:02. At 17:07 a mistyped desk number is found in the issued message. The team corrects that number, retaining verified/pending status and data time, obtains R9/A10 review/approval at 17:09/17:10, and sends the correction to D9/D10 at 17:11. It records the reason ‘prevent use of the wrong desk number’; both acknowledge at 17:13/17:14 in R06-ACK.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E2

Illustrative answer: 3

TC-E2 meets anchor 2 through verified/pending distinctions, timestamp, limited data and review/approval before release, then anchor 3 by correcting the error for prior recipients with a reason and confirmed receipt.

Anchor 3 does not require an error-free exercise; it requires all specified error-handling evidence. Ready desks do not establish how many people received service.

Anchor: Meets level 2 and, on error or an excessive-access request, sends a correction to prior recipients or offers an aggregate alternative, recording reasons and receipt.

### R07 · TC-C1
C / EOC-C

Independent fictional case, CPX time 17:30–18:00 ICT, with two complete planned opportunities and evidence cards. At 17:35 Q9 challenges an incorrect training-box edition. The team does not listen until the controller instructs it at 17:37 to hear Q9 and compare cards. It then checks a card and changes the edition in R07-D without recording why. At 17:45 Q10 requests review of a desk position because the aisle is narrow. The team listens and moves the desk but does not inspect the supplied layout or record reasons. By 18:00 there are no related-work checks, invitations to confirm shared understanding or follow-up assignments. A separate log records the 17:37 solution coaching and observations from both windows.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C1

Illustrative answer: 1

TC-C1 anchor 1: listening follows prompting and changes lack recorded reasons; complete, consistent behavior across both opportunities is absent.

Both opportunities occurred, so NO does not apply. Anchor 2 requires no solution coaching and invitation, evidence checks and reasons without blame at both events; willingness to change alone is insufficient.

Anchor: Listens after controller prompting or changes without recording reasons; consistent behavior across two opportunities is not shown.

### R08 · TC-C2
C / CPX-C

Independent fictional case, standardized CPX interval 18:10–18:40 ICT, with no solution coaching. Feedback at 18:10 identifies hard-to-read codes on training-equipment crates. At 18:14, register R08-A assigns B9 to enlarge the label font by 18:22, evidenced by a new file, and B10 to attach labels with photo evidence by 18:25. At the scheduled 18:25 review the team checks file v4 and closes the evidenced font-change action, records attachment as open, and records B11's lamination action, due 18:25 with a sample as evidence, as blocked because supplies have not arrived, matching the logs. At 18:30 the organizer supplies a fresh code-based crate-finding task to test readability. The team files the task without testing it before 18:40. All evidence is from one simulated interval, not real between-month closure.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C2

Illustrative answer: 2

TC-C2 anchor 2 is complete: owners, deadlines and evidence are specified, follow-up occurs on schedule and closed/open/blocked statuses are accurately distinguished.

Checking the revised label file is not testing whether the correction improves readability. The supplied recheck task is unused, leaving anchor 3 incomplete; this is not month 6/12 follow-up evidence.

Anchor: Assigns owner, deadline and evidence, follows up on schedule and accurately distinguishes closed, open and blocked actions.

### R09 · TC-S1
S / EOC-S

Fictional rater boundary case, CPX time 18:50–19:15 ICT. Plan R09-P specifies conflicting stair-access reports, available verification sources and deadlines for COP correction/user notification. The controller confirms all injects were delivered. After the round, every COP version, observer note and coaching log is lost from the evaluator's device. Only the plan and inject-delivery log remain. Behavior evidence cannot be recovered, and no step can be identified as performed or omitted by the team. There is no evidence that the team deleted information.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S1

Illustrative answer: NO

TC-S1 behavior evidence is insufficient because evaluator records were lost. Apply NO and document the cause and missing evidence, even though opportunity delivery is confirmed.

An inject plan does not establish team failure. Do not award 0 or reinterpret evaluator-side loss as team deletion of the COP. NO is excluded from scored denominators.

### R10 · TC-K2
K / CPX-K

Fictional rater boundary case. Document R10-S at 19:20 ICT, before the 19:30 round, declares a subset exercise in reading coordination-behavior evidence. It assigns no assessment-design or outcome–evidence explanation task and declares TC-K2 inapplicable to this scope. Scope-owner role O9 records the justification before the start. No full eight-criterion result is claimed and scope is not changed after seeing performance. This is a rater coding exercise, not a team-assessment round under the complete framework.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K2

Illustrative answer: NA

TC-K2 is outside the prespecified scope/role, with a justification owner, so NA applies. It is not retrospective coding caused by absent work.

NA is neither zero nor part of scored denominators. This subset must not be presented as a full eight-criterion team assessment. An accidentally omitted opportunity in a full round calls for considering NO, not retrospective scope revision.

## Comparison and forms

Manual check by set and criterion: eligible numeric pairs have both A/B in 0,1,2,3; exact matches have equal A/B; differences = eligible pairs minus matches. If reporting a proportion, use matches/eligible pairs and always display the denominator; zero denominator is undefined. Pairs involving NO/NA remain separate even when codes match. Blank means unentered, not NO or zero.

Use separate A/B files. prior_exposure records task/key exposure; locked_at records when ratings were fixed before seeing the other rater; score is 0–3/NO/NA. The review file retains original A/B references, scores and reasons before adding agreed_score/resolution_reason. Leave agreed_score blank for unresolved cases and record unresolved_dissent; never overwrite originals.

https://hadr-cpx.pk-research.work/en/calibration-a.csv
https://hadr-cpx.pk-research.work/en/calibration-b.csv
https://hadr-cpx.pk-research.work/en/calibration-review.csv

## Rationale sources and limits

Uses the framework/assessor guide's rules and design-rationale sources, retaining their access limits. Set design, tasks, keys and comparison tool are project syntheses, not a WHO or original-researcher training package. Ten items per set is not a reliability-study sample-size calculation.

### swiss-pharmacy-ttx-followup-2026 · Le Bloc’h et al. (2026). Tabletop Exercise for Training and Improving Business Continuity Plan Development in Hospital Pharmacies
https://doi.org/10.1017/dmp.2026.10424

Methods: Hypothesis and Outcomes; Data Collection and Analysis; Results Table 3; Limitations.

An example of later organizational team observation: five hospital pharmacies underwent FSX 7.4–10.2 months after TTX.

No baseline FSX: not proof of retention or causality, and not Thai teams, real incidents or a 12-month measurement.

### pediatric-virtual-ttx-followup-2019 · So et al. (2019). Extending the Reach of Pediatric Emergency Preparedness: A Virtual Tabletop Exercise Targeting Children’s Needs
https://journals.sagepub.com/doi/pdf/10.1177/0033354919849880

Methods: Data Sources and Data Collection / Data Analysis; Results: Follow-up Actions; Table 3; Limitations and Future Directions.

Fourteen matched respondents reported state progress at months 1 and 6, distinguishing intentions from later reported action.

Participant reports, not observed six-month team behavior; no 12-month or real-incident outcome.

### manchester-exercise-incident-transfer-2020 · Skryabina et al. (2020). The role of emergency preparedness exercises in the response to a mass casualty terrorist incident: A mixed methods study
https://pmc.ncbi.nlm.nih.gov/articles/PMC7709486/

§3.1.1–3.1.2; §4.1.2/Table 2; §4.2; §5.2 Limitations.

Accounts of using rehearsed plans, roles and coordination during a real incident within two months of exercises.

Retrospective self-report, not independent live-team scores or 6–12-month retention evidence.

### california-clinic-exercise-transfer-2010 · Fowkes et al. (2010). Exercises in Emergency Preparedness for Health Professionals in Community Clinics
https://pmc.ncbi.nlm.nih.gov/articles/PMC2952103/

Methods: Evaluation; Table 2; Results: Effects on CHCs Emergency Plans / Post Disaster Interviews; Discussion.

Three-month plan follow-up and seven post-wildfire interviews motivate separating organizational improvement from incident-use evidence.

Exercise-to-wildfire interval unspecified, with attrition and concurrent assistance; not a 6/12-month outcome.

### who-simex-2017 · WHO (2017). WHO Simulation Exercise Manual, WHO/WHE/CPI/2017.10
https://www.who.int/publications/i/item/WHO-WHE-CPI-2017.10

WHO publication landing page, Overview paragraphs 1–4 (the portion inspected in this pass).

Supports exercises as a way to practice roles and examine gaps in systems, procedures and mechanisms.

Does not endorse this KSEC rubric, follow-up schedule or cutoffs; PDF access failed in this pass.

### who-aar-2019 · WHO (2019). Guidance for after action review (AAR), WHO/WHE/CPI/2019.4
https://www.who.int/publications/i/item/WHO-WHE-CPI-2019.4

WHO publication landing page, Overview paragraphs 1–3 (the portion inspected in this pass).

Supports reviewing response for learning and improvement, with several review formats including interviews.

AAR alone is not causal evidence; this incident form is a project proposal. PDF access failed in this pass.

### ddpm-plan · กรมป้องกันและบรรเทาสาธารณภัย. แผนการป้องกันและบรรเทาสาธารณภัยแห่งชาติ พ.ศ. 2564–2570
https://www.disaster.go.th/upload/download/file_attach/3753867497001f8e1303362092706df1.pdf

Printed p.43, Figure 3–2 (PDF p.54); printed p.124, items 5.1–5.5 (PDF p.135).

Provides Thai mechanism context and information, option-development, coordination and operational-follow-up functions.

Inspected saved original-page images, not current local orders. Responsible officials must verify applicable plans and actual authority before exercise use.

Copyright 2026 PK-Research. Original source rights remain with their owners.