# Rater calibration and recheck kit · Questions only

Kit 0.1.0 · Framework 0.1.0 · 2026-10-16

This kit trains raters; it is not a real team's assessment or T0/T1/M6/M12 result. Fictional tasks and keys are project proposals, not a validated gold standard or validity/reliability-tested instrument. Raters may agree because they share a misunderstanding, so evidence and reasoning must also be reviewed. This kit supplies no passing cutoff or readiness credential.

Tasks and keys are public; collapsed answers are not secret tests. The coordinator should distribute the question-only file and separate A/B forms. Ask about and record prior exposure to tasks/keys. If a rater has seen the recheck set, prepare and review a new unseen set before using it. The old set can support learning but is not an unseen recheck.

https://hadr-cpx.pk-research.work/en/assessor-calibration/

After rater recheck, plan a team pilot and preserve independent ratings before review at each wave. This is a proposed, unvalidated plan: https://hadr-cpx.pk-research.work/en/team-pilot/

https://hadr-cpx.pk-research.work/en/team-pilot.md

## 1. Prepare materials and roles

The coordinator checks framework 0.1.0 and kit 0.1.0, supplies identical criteria/rules, and specifies raters A/B, reviewer, scope, timing, accessible format and authorized storage. Use codes instead of names; record key exposure and overlapping roles.

## 2. Rate and lock forms independently

Each rater uses their own A or B file and records 0–3/NO/NA, evidence IDs, reasons, opportunity and prompting without seeing the other rater or key. Timestamp the locked form, send it to the coordinator and retain the original. Use reasoned NO for insufficient evidence; never turn missingness into 0.

## 3. Compare pre-discussion ratings

The coordinator copies ratings from locked originals into the review form or tool below. Separate P/R and each criterion, showing matching/differing 0–3 pairs, NO/NA-related pairs and incomplete pairs. Do not create team totals, average ordinal levels or select only agreeing pairs.

## 4. Open the key and examine differences

For each item distinguish unequal evidence access, anchor interpretation, missing opportunity and transcription error. Check the teaching key against evidence rather than treating it as certification. Retain A/B originals and add resolution, reasons, reviewer and unresolved dissent; neither rank nor averaging resolves disagreement.

## 5. Recheck and plan the pilot

Use unseen R tasks, obtain fresh independent ratings and review each criterion again. Record changes, owners and recheck evidence. Before field use, seek expert review of content validity, opportunity coverage, burden and rater agreement with an appropriate sample. Prespecify ordinal analysis and uncertainty; do not claim reliability from this small kit.

## Canonical rating rules

- 0 = an observable, planned opportunity existed but the criterion behavior was not demonstrated. It does not stand for missing data, unseen behavior or absent opportunity.
- NO = not observed, insufficient evidence or no opportunity: record why and the missing opportunity. NA = inapplicable to the role/scope as declared before the round: record the justification owner. Neither is zero or enters scored denominators.
- Assign the highest anchor fully evidenced; level 3 includes level 2. Levels 2/3 require no solution coaching by controllers. Log all prompting. If the task omits a necessary opportunity, use NO with partial evidence rather than automatically awarding a low score.
- Report eight criteria grouped as K/S/E/C without a total or cross-criterion mean. Show scored/8, NO/8 and NA/8 and each criterion’s observed opportunities.
- Compare phases only for scored, comparable criterion pairs, showing pairs available/expected, roster and method changes. Do not impute missing values, turn NO/NA into zero or calculate percentage retention from ordinal 0–3 ratings.
- Report pre-consensus exact agreement by criterion, separating paired 0–3 ratings from NO/NA pairs, with counts and disagreements. During piloting, prespecify ordinal agreement statistics and uncertainty estimates with methodological expertise.

## Set P · Practice and discussion

Rate independently before seeing the key, then check reasoning against anchors and discuss differences. Do not substitute post-discussion ratings for independent ones.

### P01 · TC-K1
K / EOC-K

Independent fictional case, CPX time 08:10–08:30 ICT, with no solution coaching. Organizer-checked exercise-only scope card Z, edition 3, clauses 6–8, assigns N to propose options, B to coordinate and D to approve simulated room use in the morning only. At 08:16, table P01-A and the team's explanation cover all three roles, cite the edition/clauses and identify afternoon permission as unknown. At 08:20 the controller changes the request to afternoon use, offering scope verification by V and referral through B. The team attaches the new card but does not explain decision implications, identify verification/referral roles or revise the table before the window ends. These cards grant no real authority.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P02 · TC-K2
K / CPX-K

Independent fictional case, CPX time 08:40–09:00 ICT, with no solution coaching. The team receives observation record O7 of a map-version check, confidence response S7 and invented incident-record facsimile I7. It must link the current-map outcome to events and evidence and explain limits. At 08:47, map P02-M links receipt of a new map to O7's observed version check, but classifies S7's ‘confident I use the latest edition’ as direct observation, leaves I7 unlinked and all limits blank. At 08:52 a card reveals that I7 lacks a version number. Time to examine it and request further evidence lasts until 09:00; no explanation is added.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P03 · TC-S1
S / EOC-S

Independent fictional case, CPX time 09:10–09:35 ICT, with no solution coaching. Log P03-L retains channel A's 09:09 report that the simulated bridge is open and channel B's 09:12 report that it is closed, recording sources, times, conflicting/pending status and verifier V7. At 09:18 V7 supplies a closure-confirmation card. The team retains COP v1 and issues v2 at 09:20, linking V7's reply before the 09:22 deadline. At 09:24 it corrects the SITREP and route decision to avoid the bridge and sends the correction to simulated desks U7/U8, which used v1. Both acknowledge v2 in P03-ACK at 09:27/09:29, before the 09:32 deadline.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P04 · TC-S2
S / EOC-S

Independent fictional case, CPX time 09:45–10:10 ICT, with no solution coaching. The team must send a training-radio check request to desk T7, available to acknowledge, follow up at 09:55 and hand over at 10:05. The agreement permits referral to T8 if T7 cannot perform it. At 09:49, notebook P04-Q records a general-board post saying ‘check radios’ without a receiver, marked complete because it was posted. The observer sees no receipt check or 09:55 follow-up despite the available receiving desk. At 09:58 a card reports T7 lacks equipment and supplies the T8 route. The team takes no action and removes the task from the 10:05 handover.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P05 · TC-E1
E / EOC-E

Independent fictional case, CPX time 10:20–10:45 ICT, with no solution coaching. Twelve training battery packs are available. Nearby H needs eight; uphill J needs eight and reports backup power nearly exhausted. At 10:27, table P05-O compares H8/J4, easier delivery but four waiting at J, with H4/J8, reducing J's depletion risk but leaving four waiting at H and taking longer. It chooses the latter, marks J's remaining-power duration unverified and assigns F7 to follow up both sites at 10:38. At 10:32 J's simulated representative challenges connector compatibility; an inspection card is available. The team records the challenge but does not check it, justify a revised/retained choice, plan mitigation or schedule review before the window ends.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P06 · TC-E2
E / EOC-E

Independent fictional case, CPX time 10:55–11:20 ICT, with no solution coaching. Evidence confirms four simulated charging points ready and two pending inspection. Reviewer R7 and approver A7 must act before release; both are available from 10:55 to 11:05. At 11:03 the team changes draft P06-B from ‘all points ready’ to ‘four confirmed ready’ and removes fictional household codes. The 11:05 issued message omits the data time and two pending points and has no review/approval record. At 11:10 an excessive household-detail request arrives with a channel for an aggregate reply. The team sends no response before the window ends.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P07 · TC-C1
C / EOC-C

Independent fictional case, CPX time 11:30–12:00 ICT, with no solution coaching and two planned opportunities. At 11:35 Q7 challenges an outdated opening time. The team invites explanation, compares time cards and records in P07-D a change to 12:30 because the latest card supersedes the earlier one, without blame. At 11:43 Q8 requests review of communication for people unable to read. The team invites evidence, checks the accessibility card and records retaining both audio and text because they serve two groups, without blame. At 11:48 it checks related work and corrects an announcement schedule still using the old time. Q7/Q8 are invited to read back the reasoning and confirm understanding. Audio reach remains unresolved; V8 owns verification, and P07-F at 11:56 records follow-up, a pending layout and the next review at 12:05.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P08 · TC-C2
C / CPX-C

Independent fictional case, standardized CPX interval 12:10–12:40 ICT, with no solution coaching. At 12:10 feedback says the training map legend does not distinguish open from pending-inspection points. An editable form is available, with follow-up at 12:25 and a fresh map task to test the correction at 12:30. Complete observation record P08-N shows acknowledgment of feedback but no action entry, assignment or artifact revision. The team does not return to the scheduled 12:25 review despite the reserved window, or use the 12:30 recheck task. The unchanged map and empty register are retained. This is simulated time, not month 6/12 follow-up.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P09 · TC-C1
C / EOC-C

Fictional rater boundary case, CPX time 13:00–13:20 ICT, with no solution coaching. Plan P09-P schedules a report challenge at 13:05 and decision review at 13:15. At the first, the team invites the challenger, checks a room-capacity card and records a reasoned count revision without blame; evidence is complete. The organizer ends the window before delivering the second event because of a timetable error. The controller log confirms the second opportunity was never supplied, rather than refused by the team. No second-opportunity behavior evidence is available.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### P10 · TC-E2
E / EOC-E

Fictional rater boundary case. Document P10-S, timestamped 13:30 ICT before the 13:40 exercise, limits this subset practice to internal handover evidence for raters. It assigns no message drafting, review, approval or release roles and declares TC-E2 outside scope. Simulated scope owner O7 records that justification before the round; scope is not changed retrospectively. The activity is explicitly a rater subset exercise, not a full eight-criterion team assessment.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

## Set R · Recheck with new tasks

Use after calibration, with raters who have not seen set R. Rate separately again before opening the key. P/R difficulty equivalence has not been established, so increased agreement alone does not establish an effect of calibration.

### R01 · TC-K1
K / EOC-K

Independent fictional rater recheck case, CPX time 14:00–14:25 ICT, with no solution coaching. Organizer-checked exercise card Y, edition 2, clauses 11–13, assigns J to propose, M to coordinate and A9 to approve borrowing no more than six simulated tables. At 14:06, table R01-T explains all three roles, cites edition/clauses and marks scope above six as unknown. At 14:10 the request changes to nine. The team explains that A9 cannot yet approve this request. At 14:14, v2 assigns V9 to check scope and M to refer to role X9 under the card, explaining that scope must be verified before proposing an over-limit request. The card operates only within the simulation and grants no real authority.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R02 · TC-K2
K / CPX-K

Independent fictional case, CPX time 14:35–15:00 ICT, with no solution coaching. At 14:44, map R02-M links the pre-release message-review outcome to all three events and artifacts: CPX log O9 records review in this task only; post-training response S9 indicates confidence, not observed behavior; invented incident-record facsimile I9 links its release event to recorded review activity without proving training caused the result. At 14:48 a new card reveals I9's review time is after release. The controller makes further records available on request until 15:00. The team attaches the card but does not identify a conclusion to withhold or additional evidence answering the discrepancy.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R03 · TC-S1
S / EOC-S

Independent fictional case, CPX time 15:10–15:35 ICT, with no solution coaching. Simulated store report A at 15:11 gives seven training pumps ready; B at 15:13 gives five. An evidence-based COP update is due by 15:25 and notice to prior users by 15:32; verification sources remain available. At 15:18 the team keeps both cards and writes ‘5 or 7, needs checking’ in R03-C, but omits sources/times from the row and does not identify a verification owner. The row remains unchanged at the end, with no verification request, revised SITREP or notice to prior users. The observer retains all cards and notes.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R04 · TC-S2
S / EOC-S

Independent fictional case, CPX time 15:45–16:10 ICT, with no solution coaching. At 15:48 request R04-Q assigns W9 a training-sign inventory, status draft pending, reply due 15:55 and a fictional-codes-only constraint. W9 accurately reads back all fields at 15:49. The team follows up at 15:55 and records the draft still pending. At 15:58 W9 reports the simulated printer unavailable. The exercise agreement permits referral to available W10. The team retains the notice but neither refers nor confirms replacement ownership. The 16:05 handover retains the open W9 task and disruption; the incoming role reads it back correctly.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R05 · TC-E1
E / EOC-E

Independent fictional case, CPX time 16:20–16:45 ICT, with no solution coaching. Six simulated quiet-room slots are available. Nearby group U needs four; group V on a less accessible floor needs four and has a card describing noise-related access constraints. At 16:27, table R05-A assigns all six slots to U because scheduling is easy, without mentioning V, people waiting or adverse effects. At 16:32 V's simulated representative supplies an access card and requests review. Evidence checking and comparison of two options are available until 16:45. The observer records the team filing the card unread and continuing the same schedule; waiting-group/adverse-effect fields remain blank.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R06 · TC-E2
E / EOC-E

Independent fictional case, CPX time 16:55–17:20 ICT, with no solution coaching. Draft R06-B states ‘As of 16:55, three simulated service desks confirmed ready; one awaiting inspection; not a count of people served,’ with no identifying data. R9 checks it against evidence cards at 16:58 and A10 gives simulated approval at 17:00 before release to D9/D10 at 17:02. At 17:07 a mistyped desk number is found in the issued message. The team corrects that number, retaining verified/pending status and data time, obtains R9/A10 review/approval at 17:09/17:10, and sends the correction to D9/D10 at 17:11. It records the reason ‘prevent use of the wrong desk number’; both acknowledge at 17:13/17:14 in R06-ACK.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-E2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R07 · TC-C1
C / EOC-C

Independent fictional case, CPX time 17:30–18:00 ICT, with two complete planned opportunities and evidence cards. At 17:35 Q9 challenges an incorrect training-box edition. The team does not listen until the controller instructs it at 17:37 to hear Q9 and compare cards. It then checks a card and changes the edition in R07-D without recording why. At 17:45 Q10 requests review of a desk position because the aisle is narrow. The team listens and moves the desk but does not inspect the supplied layout or record reasons. By 18:00 there are no related-work checks, invitations to confirm shared understanding or follow-up assignments. A separate log records the 17:37 solution coaching and observations from both windows.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R08 · TC-C2
C / CPX-C

Independent fictional case, standardized CPX interval 18:10–18:40 ICT, with no solution coaching. Feedback at 18:10 identifies hard-to-read codes on training-equipment crates. At 18:14, register R08-A assigns B9 to enlarge the label font by 18:22, evidenced by a new file, and B10 to attach labels with photo evidence by 18:25. At the scheduled 18:25 review the team checks file v4 and closes the evidenced font-change action, records attachment as open, and records B11's lamination action, due 18:25 with a sample as evidence, as blocked because supplies have not arrived, matching the logs. At 18:30 the organizer supplies a fresh code-based crate-finding task to test readability. The team files the task without testing it before 18:40. All evidence is from one simulated interval, not real between-month closure.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-C2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R09 · TC-S1
S / EOC-S

Fictional rater boundary case, CPX time 18:50–19:15 ICT. Plan R09-P specifies conflicting stair-access reports, available verification sources and deadlines for COP correction/user notification. The controller confirms all injects were delivered. After the round, every COP version, observer note and coaching log is lost from the evaluator's device. Only the plan and inject-delivery log remain. Behavior evidence cannot be recovered, and no step can be identified as performed or omitted by the team. There is no evidence that the team deleted information.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-S1

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

### R10 · TC-K2
K / CPX-K

Fictional rater boundary case. Document R10-S at 19:20 ICT, before the 19:30 round, declares a subset exercise in reading coordination-behavior evidence. It assigns no assessment-design or outcome–evidence explanation task and declares TC-K2 inapplicable to this scope. Scope-owner role O9 records the justification before the start. No full eight-criterion result is claimed and scope is not changed after seeing performance. This is a rater coding exercise, not a team-assessment round under the complete framework.

https://hadr-cpx.pk-research.work/en/team-assessment/#TC-K2

Rater code / rating / time and evidence / reason / opportunity / prompting / form lock time: __________________

## Comparison and forms

Manual check by set and criterion: eligible numeric pairs have both A/B in 0,1,2,3; exact matches have equal A/B; differences = eligible pairs minus matches. If reporting a proportion, use matches/eligible pairs and always display the denominator; zero denominator is undefined. Pairs involving NO/NA remain separate even when codes match. Blank means unentered, not NO or zero.

Use separate A/B files. prior_exposure records task/key exposure; locked_at records when ratings were fixed before seeing the other rater; score is 0–3/NO/NA. The review file retains original A/B references, scores and reasons before adding agreed_score/resolution_reason. Leave agreed_score blank for unresolved cases and record unresolved_dissent; never overwrite originals.

https://hadr-cpx.pk-research.work/en/calibration-a.csv
https://hadr-cpx.pk-research.work/en/calibration-b.csv
https://hadr-cpx.pk-research.work/en/calibration-review.csv

## Rationale sources and limits

Uses the framework/assessor guide's rules and design-rationale sources, retaining their access limits. Set design, tasks, keys and comparison tool are project syntheses, not a WHO or original-researcher training package. Ten items per set is not a reliability-study sample-size calculation.

### swiss-pharmacy-ttx-followup-2026 · Le Bloc’h et al. (2026). Tabletop Exercise for Training and Improving Business Continuity Plan Development in Hospital Pharmacies
https://doi.org/10.1017/dmp.2026.10424

Methods: Hypothesis and Outcomes; Data Collection and Analysis; Results Table 3; Limitations.

An example of later organizational team observation: five hospital pharmacies underwent FSX 7.4–10.2 months after TTX.

No baseline FSX: not proof of retention or causality, and not Thai teams, real incidents or a 12-month measurement.

### pediatric-virtual-ttx-followup-2019 · So et al. (2019). Extending the Reach of Pediatric Emergency Preparedness: A Virtual Tabletop Exercise Targeting Children’s Needs
https://journals.sagepub.com/doi/pdf/10.1177/0033354919849880

Methods: Data Sources and Data Collection / Data Analysis; Results: Follow-up Actions; Table 3; Limitations and Future Directions.

Fourteen matched respondents reported state progress at months 1 and 6, distinguishing intentions from later reported action.

Participant reports, not observed six-month team behavior; no 12-month or real-incident outcome.

### manchester-exercise-incident-transfer-2020 · Skryabina et al. (2020). The role of emergency preparedness exercises in the response to a mass casualty terrorist incident: A mixed methods study
https://pmc.ncbi.nlm.nih.gov/articles/PMC7709486/

§3.1.1–3.1.2; §4.1.2/Table 2; §4.2; §5.2 Limitations.

Accounts of using rehearsed plans, roles and coordination during a real incident within two months of exercises.

Retrospective self-report, not independent live-team scores or 6–12-month retention evidence.

### california-clinic-exercise-transfer-2010 · Fowkes et al. (2010). Exercises in Emergency Preparedness for Health Professionals in Community Clinics
https://pmc.ncbi.nlm.nih.gov/articles/PMC2952103/

Methods: Evaluation; Table 2; Results: Effects on CHCs Emergency Plans / Post Disaster Interviews; Discussion.

Three-month plan follow-up and seven post-wildfire interviews motivate separating organizational improvement from incident-use evidence.

Exercise-to-wildfire interval unspecified, with attrition and concurrent assistance; not a 6/12-month outcome.

### who-simex-2017 · WHO (2017). WHO Simulation Exercise Manual, WHO/WHE/CPI/2017.10
https://www.who.int/publications/i/item/WHO-WHE-CPI-2017.10

WHO publication landing page, Overview paragraphs 1–4 (the portion inspected in this pass).

Supports exercises as a way to practice roles and examine gaps in systems, procedures and mechanisms.

Does not endorse this KSEC rubric, follow-up schedule or cutoffs; PDF access failed in this pass.

### who-aar-2019 · WHO (2019). Guidance for after action review (AAR), WHO/WHE/CPI/2019.4
https://www.who.int/publications/i/item/WHO-WHE-CPI-2019.4

WHO publication landing page, Overview paragraphs 1–3 (the portion inspected in this pass).

Supports reviewing response for learning and improvement, with several review formats including interviews.

AAR alone is not causal evidence; this incident form is a project proposal. PDF access failed in this pass.

### ddpm-plan · กรมป้องกันและบรรเทาสาธารณภัย. แผนการป้องกันและบรรเทาสาธารณภัยแห่งชาติ พ.ศ. 2564–2570
https://www.disaster.go.th/upload/download/file_attach/3753867497001f8e1303362092706df1.pdf

Printed p.43, Figure 3–2 (PDF p.54); printed p.124, items 5.1–5.5 (PDF p.135).

Provides Thai mechanism context and information, option-development, coordination and operational-follow-up functions.

Inspected saved original-page images, not current local orders. Responsible officials must verify applicable plans and actual authority before exercise use.

Copyright 2026 PK-Research. Original source rights remain with their owners.