Proposed pilot · Kit v0.1.0 · 16 October 2026

Rater calibration and recheck kit

Two new ten-item sets, independent A/B rating forms and a pre-discussion comparison tool linked to all eight team CPX criteria and KSEC/OBE.

This kit trains raters; it is not a real team's assessment or T0/T1/M6/M12 result. Fictional tasks and keys are project proposals, not a validated gold standard or validity/reliability-tested instrument. Raters may agree because they share a misunderstanding, so evidence and reasoning must also be reviewed. This kit supplies no passing cutoff or readiness credential.

Tasks and keys are public; collapsed answers are not secret tests. The coordinator should distribute the question-only file and separate A/B forms. Ask about and record prior exposure to tasks/keys. If a rater has seen the recheck set, prepare and review a new unseen set before using it. The old set can support learning but is not an unseen recheck.

After rater recheck, plan the team pilot and its decision points
Preserve original and reviewed ratings separately in baseline, post and 6/12-month forms

From independent ratings to recheck

1. Prepare materials and roles

The coordinator checks framework 0.1.0 and kit 0.1.0, supplies identical criteria/rules, and specifies raters A/B, reviewer, scope, timing, accessible format and authorized storage. Use codes instead of names; record key exposure and overlapping roles.

2. Rate and lock forms independently

Each rater uses their own A or B file and records 0–3/NO/NA, evidence IDs, reasons, opportunity and prompting without seeing the other rater or key. Timestamp the locked form, send it to the coordinator and retain the original. Use reasoned NO for insufficient evidence; never turn missingness into 0.

3. Compare pre-discussion ratings

The coordinator copies ratings from locked originals into the review form or tool below. Separate P/R and each criterion, showing matching/differing 0–3 pairs, NO/NA-related pairs and incomplete pairs. Do not create team totals, average ordinal levels or select only agreeing pairs.

4. Open the key and examine differences

For each item distinguish unequal evidence access, anchor interpretation, missing opportunity and transcription error. Check the teaching key against evidence rather than treating it as certification. Retain A/B originals and add resolution, reasons, reviewer and unresolved dissent; neither rank nor averaging resolves disagreement.

5. Recheck and plan the pilot

Use unseen R tasks, obtain fresh independent ratings and review each criterion again. Record changes, owners and recheck evidence. Before field use, seek expert review of content validity, opportunity coverage, burden and rater agreement with an appropriate sample. Prespecify ordinal analysis and uncertainty; do not claim reliability from this small kit.

Review canonical 0–3 / NO / NA rules
  • 0 = an observable, planned opportunity existed but the criterion behavior was not demonstrated. It does not stand for missing data, unseen behavior or absent opportunity.
  • NO = not observed, insufficient evidence or no opportunity: record why and the missing opportunity. NA = inapplicable to the role/scope as declared before the round: record the justification owner. Neither is zero or enters scored denominators.
  • Assign the highest anchor fully evidenced; level 3 includes level 2. Levels 2/3 require no solution coaching by controllers. Log all prompting. If the task omits a necessary opportunity, use NO with partial evidence rather than automatically awarding a low score.
  • Report eight criteria grouped as K/S/E/C without a total or cross-criterion mean. Show scored/8, NO/8 and NA/8 and each criterion’s observed opportunities.
  • Compare phases only for scored, comparable criterion pairs, showing pairs available/expected, roster and method changes. Do not impute missing values, turn NO/NA into zero or calculate percentage retention from ordinal 0–3 ratings.
  • Report pre-consensus exact agreement by criterion, separating paired 0–3 ratings from NO/NA pairs, with counts and disagreements. During piloting, prespecify ordinal agreement statistics and uncertainty estimates with methodological expertise.

Set P · Practice and discussion

Rate independently before seeing the key, then check reasoning against anchors and discuss differences. Do not substitute post-discussion ratings for independent ones.

P01 · TC-K1 · Explain roles and authority boundaries

K / EOC-K · Open canonical criterion

Independent fictional case, CPX time 08:10–08:30 ICT, with no solution coaching. Organizer-checked exercise-only scope card Z, edition 3, clauses 6–8, assigns N to propose options, B to coordinate and D to approve simulated room use in the morning only. At 08:16, table P01-A and the team's explanation cover all three roles, cite the edition/clauses and identify afternoon permission as unknown. At 08:20 the controller changes the request to afternoon use, offering scope verification by V and referral through B. The team attaches the new card but does not explain decision implications, identify verification/referral roles or revise the table before the window ends. These cards grant no real authority.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 2

TC-K1 anchor 2 is complete: proposal, coordination and approval roles, organizer-checked edition/clauses and unknowns are explained without solution coaching.

A changed-condition opportunity was supplied, but decision implications, verification/referral roles and a justified table revision are absent, so anchor 3 is incomplete; this is not NO.

P02 · TC-K2 · Explain what an evidence item establishes

K / CPX-K · Open canonical criterion

Independent fictional case, CPX time 08:40–09:00 ICT, with no solution coaching. The team receives observation record O7 of a map-version check, confidence response S7 and invented incident-record facsimile I7. It must link the current-map outcome to events and evidence and explain limits. At 08:47, map P02-M links receipt of a new map to O7's observed version check, but classifies S7's ‘confident I use the latest edition’ as direct observation, leaves I7 unlinked and all limits blank. At 08:52 a card reveals that I7 lacks a version number. Time to examine it and request further evidence lasts until 09:00; no explanation is added.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 1

TC-K2 anchor 1: one evidence type is linked to the outcome, but self-report is confused with observation and limits are omitted.

Anchor 2 requires complete task links and distinctions among all three evidence types with limits. Materials and explanation opportunities were available, so NO does not apply.

P03 · TC-S1 · Update the situation picture with a verification trail

S / EOC-S · Open canonical criterion

Independent fictional case, CPX time 09:10–09:35 ICT, with no solution coaching. Log P03-L retains channel A's 09:09 report that the simulated bridge is open and channel B's 09:12 report that it is closed, recording sources, times, conflicting/pending status and verifier V7. At 09:18 V7 supplies a closure-confirmation card. The team retains COP v1 and issues v2 at 09:20, linking V7's reply before the 09:22 deadline. At 09:24 it corrects the SITREP and route decision to avoid the bridge and sends the correction to simulated desks U7/U8, which used v1. Both acknowledge v2 in P03-ACK at 09:27/09:29, before the 09:32 deadline.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 3

TC-S1 meets anchor 2 through retained conflicts, provenance, times, status, verification ownership and a timely versioned update, then anchor 3 through SITREP/decision correction, notification of old-version users and confirmed receipt.

Receipts from both prior users distinguish this from merely sending a correction. The rating concerns simulated work and establishes no real bridge condition.

P04 · TC-S2 · Close the loop on task coordination

S / EOC-S · Open canonical criterion

Independent fictional case, CPX time 09:45–10:10 ICT, with no solution coaching. The team must send a training-radio check request to desk T7, available to acknowledge, follow up at 09:55 and hand over at 10:05. The agreement permits referral to T8 if T7 cannot perform it. At 09:49, notebook P04-Q records a general-board post saying ‘check radios’ without a receiver, marked complete because it was posted. The observer sees no receipt check or 09:55 follow-up despite the available receiving desk. At 09:58 a card reports T7 lacks equipment and supplies the T8 route. The team takes no action and removes the task from the 10:05 handover.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 0

TC-S2 anchor 0: a task and receiving role were available, but no receiver is identified and transmission is treated as completion without a receipt check. This behavior was observed.

Even anchor 1's dispatch to an identified role is absent. This is neither missing data nor an absent opportunity, so NO does not apply.

P05 · TC-E1 · Justify allocation with attention to those still waiting

E / EOC-E · Open canonical criterion

Independent fictional case, CPX time 10:20–10:45 ICT, with no solution coaching. Twelve training battery packs are available. Nearby H needs eight; uphill J needs eight and reports backup power nearly exhausted. At 10:27, table P05-O compares H8/J4, easier delivery but four waiting at J, with H4/J8, reducing J's depletion risk but leaving four waiting at H and taking longer. It chooses the latter, marks J's remaining-power duration unverified and assigns F7 to follow up both sites at 10:38. At 10:32 J's simulated representative challenges connector compatibility; an inspection card is available. The team records the challenge but does not check it, justify a revised/retained choice, plan mitigation or schedule review before the window ends.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 2

TC-E1 anchor 2 is complete: two options are compared by need/access, with recipients, waiting groups, reasons, uncertainty and follow-up ownership, without solution coaching.

Challenge and verification opportunities existed, but evidence checking, a reasoned response, mitigation and review required for anchor 3 are absent. This is not real allocation advice.

P06 · TC-E2 · Communicate within evidence and data limits

E / EOC-E · Open canonical criterion

Independent fictional case, CPX time 10:55–11:20 ICT, with no solution coaching. Evidence confirms four simulated charging points ready and two pending inspection. Reviewer R7 and approver A7 must act before release; both are available from 10:55 to 11:05. At 11:03 the team changes draft P06-B from ‘all points ready’ to ‘four confirmed ready’ and removes fictional household codes. The 11:05 issued message omits the data time and two pending points and has no review/approval record. At 11:10 an excessive household-detail request arrives with a channel for an aggregate reply. The team sends no response before the window ends.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 1

TC-E2 anchor 1: data and a claim are reduced, but time, uncertainty and the agreed review step remain absent.

The revision does not meet anchor 2's verified/pending distinction, timestamp and recorded review/approval before release. Review and request-response opportunities were supplied.

P07 · TC-C1 · Hear challenges and explain revisions

C / EOC-C · Open canonical criterion

Independent fictional case, CPX time 11:30–12:00 ICT, with no solution coaching and two planned opportunities. At 11:35 Q7 challenges an outdated opening time. The team invites explanation, compares time cards and records in P07-D a change to 12:30 because the latest card supersedes the earlier one, without blame. At 11:43 Q8 requests review of communication for people unable to read. The team invites evidence, checks the accessibility card and records retaining both audio and text because they serve two groups, without blame. At 11:48 it checks related work and corrects an announcement schedule still using the old time. Q7/Q8 are invited to read back the reasoning and confirm understanding. Audio reach remains unresolved; V8 owns verification, and P07-F at 11:56 records follow-up, a pending layout and the next review at 12:05.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 3

TC-C1 meets anchor 2 at both opportunities through invitation, evidence checks and reasons without blame, then anchor 3 through related-work checks, challenger confirmation of understanding and evidenced follow-up of the unresolved concern.

Follow-up is evidenced even though the concern remains unresolved. This rates behavior across two opportunities, not personality or successful resolution of audio reach.

P08 · TC-C2 · Own improvements through recheck

C / CPX-C · Open canonical criterion

Independent fictional case, standardized CPX interval 12:10–12:40 ICT, with no solution coaching. At 12:10 feedback says the training map legend does not distinguish open from pending-inspection points. An editable form is available, with follow-up at 12:25 and a fresh map task to test the correction at 12:30. Complete observation record P08-N shows acknowledgment of feedback but no action entry, assignment or artifact revision. The team does not return to the scheduled 12:25 review despite the reserved window, or use the 12:30 recheck task. The unchanged map and empty register are retained. This is simulated time, not month 6/12 follow-up.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 0

TC-C2 anchor 0: feedback, a follow-up window and a recheck task existed, but the team was observed creating no action. Evidence was not lost.

Even anchor 1's listed improvement is absent. NO does not replace observed nonperformance with an opportunity, and no long-term retention inference follows.

P09 · TC-C1 · Hear challenges and explain revisions

C / EOC-C · Open canonical criterion

Fictional rater boundary case, CPX time 13:00–13:20 ICT, with no solution coaching. Plan P09-P schedules a report challenge at 13:05 and decision review at 13:15. At the first, the team invites the challenger, checks a room-capacity card and records a reasoned count revision without blame; evidence is complete. The organizer ends the window before delivering the second event because of a timetable error. The controller log confirms the second opportunity was never supplied, rather than refused by the team. No second-opportunity behavior evidence is available.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: NO

TC-C1 requires at least two planned opportunities, but the necessary second opportunity was not supplied. Apply NO, documenting why and retaining the partial evidence from the first event.

Do not award 0 or 1 for the organizer's omitted opportunity, or 2 from one event. NO is excluded from scored denominators; another opportunity is needed for a complete criterion judgment.

P10 · TC-E2 · Communicate within evidence and data limits

E / EOC-E · Open canonical criterion

Fictional rater boundary case. Document P10-S, timestamped 13:30 ICT before the 13:40 exercise, limits this subset practice to internal handover evidence for raters. It assigns no message drafting, review, approval or release roles and declares TC-E2 outside scope. Simulated scope owner O7 records that justification before the round; scope is not changed retrospectively. The activity is explicitly a rater subset exercise, not a full eight-criterion team assessment.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: NA

TC-E2 is inapplicable to the role/scope declared before the round, with a recorded justification owner, meeting the NA rule.

NA is neither zero nor part of scored denominators. This teaches a scope boundary only; it cannot be presented as a full eight-criterion team assessment or used to hide an accidentally omitted opportunity.

Set R · Recheck with new tasks

Use after calibration, with raters who have not seen set R. Rate separately again before opening the key. P/R difficulty equivalence has not been established, so increased agreement alone does not establish an effect of calibration.

R01 · TC-K1 · Explain roles and authority boundaries

K / EOC-K · Open canonical criterion

Independent fictional rater recheck case, CPX time 14:00–14:25 ICT, with no solution coaching. Organizer-checked exercise card Y, edition 2, clauses 11–13, assigns J to propose, M to coordinate and A9 to approve borrowing no more than six simulated tables. At 14:06, table R01-T explains all three roles, cites edition/clauses and marks scope above six as unknown. At 14:10 the request changes to nine. The team explains that A9 cannot yet approve this request. At 14:14, v2 assigns V9 to check scope and M to refer to role X9 under the card, explaining that scope must be verified before proposing an over-limit request. The card operates only within the simulation and grants no real authority.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 3

TC-K1 meets anchor 2 through roles, edition/clauses and unknowns, then anchor 3 through changed-condition implications, verification/referral roles and a justified revised table.

Evidence goes beyond recording a pending check: decision implications and a revised role route are explicit. It does not establish that the over-limit request was approved.

R02 · TC-K2 · Explain what an evidence item establishes

K / CPX-K · Open canonical criterion

Independent fictional case, CPX time 14:35–15:00 ICT, with no solution coaching. At 14:44, map R02-M links the pre-release message-review outcome to all three events and artifacts: CPX log O9 records review in this task only; post-training response S9 indicates confidence, not observed behavior; invented incident-record facsimile I9 links its release event to recorded review activity without proving training caused the result. At 14:48 a new card reveals I9's review time is after release. The controller makes further records available on request until 15:00. The team attaches the card but does not identify a conclusion to withhold or additional evidence answering the discrepancy.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 2

TC-K2 anchor 2 is complete: all outcome–event–artifact links and distinctions among observation, self-report and incident evidence, with each type's limits, are supplied without solution coaching.

After the conflict appears, unsupported conclusions and question-specific additional evidence required for anchor 3 are absent. Attaching the card does not explain its implications for the new conclusion.

R03 · TC-S1 · Update the situation picture with a verification trail

S / EOC-S · Open canonical criterion

Independent fictional case, CPX time 15:10–15:35 ICT, with no solution coaching. Simulated store report A at 15:11 gives seven training pumps ready; B at 15:13 gives five. An evidence-based COP update is due by 15:25 and notice to prior users by 15:32; verification sources remain available. At 15:18 the team keeps both cards and writes ‘5 or 7, needs checking’ in R03-C, but omits sources/times from the row and does not identify a verification owner. The row remains unchanged at the end, with no verification request, revised SITREP or notice to prior users. The observer retains all cards and notes.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 1

TC-S1 anchor 1: doubt and pending verification are recorded, but provenance, time and verification ownership are missing.

Retaining cards and marking them for checking do not complete anchor 2’s verification assignment and timely evidence-based versioning. Opportunities and records are complete; NO does not apply.

R04 · TC-S2 · Close the loop on task coordination

S / EOC-S · Open canonical criterion

Independent fictional case, CPX time 15:45–16:10 ICT, with no solution coaching. At 15:48 request R04-Q assigns W9 a training-sign inventory, status draft pending, reply due 15:55 and a fictional-codes-only constraint. W9 accurately reads back all fields at 15:49. The team follows up at 15:55 and records the draft still pending. At 15:58 W9 reports the simulated printer unavailable. The exercise agreement permits referral to available W10. The team retains the notice but neither refers nor confirms replacement ownership. The 16:05 handover retains the open W9 task and disruption; the incoming role reads it back correctly.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 2

TC-S2 anchor 2 is complete: owner, status, deadline and constraints are sent, accurately read back and followed up on schedule, without solution coaching.

Retaining open work at handover is insufficient for anchor 3 because the resource failure is not followed by available rerouting and confirmed replacement ownership.

R05 · TC-E1 · Justify allocation with attention to those still waiting

E / EOC-E · Open canonical criterion

Independent fictional case, CPX time 16:20–16:45 ICT, with no solution coaching. Six simulated quiet-room slots are available. Nearby group U needs four; group V on a less accessible floor needs four and has a card describing noise-related access constraints. At 16:27, table R05-A assigns all six slots to U because scheduling is easy, without mentioning V, people waiting or adverse effects. At 16:32 V's simulated representative supplies an access card and requests review. Evidence checking and comparison of two options are available until 16:45. The observer records the team filing the card unread and continuing the same schedule; waiting-group/adverse-effect fields remain blank.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 0

TC-E1 anchor 0: evidence of an unreached group and a review opportunity exist, but the team omits people waiting and adverse consequences of allocation.

Even anchor 1's mention of those waiting is absent. Records show omission despite opportunity, not lost evidence; the case is not a real allocation proposal.

R06 · TC-E2 · Communicate within evidence and data limits

E / EOC-E · Open canonical criterion

Independent fictional case, CPX time 16:55–17:20 ICT, with no solution coaching. Draft R06-B states ‘As of 16:55, three simulated service desks confirmed ready; one awaiting inspection; not a count of people served,’ with no identifying data. R9 checks it against evidence cards at 16:58 and A10 gives simulated approval at 17:00 before release to D9/D10 at 17:02. At 17:07 a mistyped desk number is found in the issued message. The team corrects that number, retaining verified/pending status and data time, obtains R9/A10 review/approval at 17:09/17:10, and sends the correction to D9/D10 at 17:11. It records the reason ‘prevent use of the wrong desk number’; both acknowledge at 17:13/17:14 in R06-ACK.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 3

TC-E2 meets anchor 2 through verified/pending distinctions, timestamp, limited data and review/approval before release, then anchor 3 by correcting the error for prior recipients with a reason and confirmed receipt.

Anchor 3 does not require an error-free exercise; it requires all specified error-handling evidence. Ready desks do not establish how many people received service.

R07 · TC-C1 · Hear challenges and explain revisions

C / EOC-C · Open canonical criterion

Independent fictional case, CPX time 17:30–18:00 ICT, with two complete planned opportunities and evidence cards. At 17:35 Q9 challenges an incorrect training-box edition. The team does not listen until the controller instructs it at 17:37 to hear Q9 and compare cards. It then checks a card and changes the edition in R07-D without recording why. At 17:45 Q10 requests review of a desk position because the aisle is narrow. The team listens and moves the desk but does not inspect the supplied layout or record reasons. By 18:00 there are no related-work checks, invitations to confirm shared understanding or follow-up assignments. A separate log records the 17:37 solution coaching and observations from both windows.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 1

TC-C1 anchor 1: listening follows prompting and changes lack recorded reasons; complete, consistent behavior across both opportunities is absent.

Both opportunities occurred, so NO does not apply. Anchor 2 requires no solution coaching and invitation, evidence checks and reasons without blame at both events; willingness to change alone is insufficient.

R08 · TC-C2 · Own improvements through recheck

C / CPX-C · Open canonical criterion

Independent fictional case, standardized CPX interval 18:10–18:40 ICT, with no solution coaching. Feedback at 18:10 identifies hard-to-read codes on training-equipment crates. At 18:14, register R08-A assigns B9 to enlarge the label font by 18:22, evidenced by a new file, and B10 to attach labels with photo evidence by 18:25. At the scheduled 18:25 review the team checks file v4 and closes the evidenced font-change action, records attachment as open, and records B11's lamination action, due 18:25 with a sample as evidence, as blocked because supplies have not arrived, matching the logs. At 18:30 the organizer supplies a fresh code-based crate-finding task to test readability. The team files the task without testing it before 18:40. All evidence is from one simulated interval, not real between-month closure.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: 2

TC-C2 anchor 2 is complete: owners, deadlines and evidence are specified, follow-up occurs on schedule and closed/open/blocked statuses are accurately distinguished.

Checking the revised label file is not testing whether the correction improves readability. The supplied recheck task is unused, leaving anchor 3 incomplete; this is not month 6/12 follow-up evidence.

R09 · TC-S1 · Update the situation picture with a verification trail

S / EOC-S · Open canonical criterion

Fictional rater boundary case, CPX time 18:50–19:15 ICT. Plan R09-P specifies conflicting stair-access reports, available verification sources and deadlines for COP correction/user notification. The controller confirms all injects were delivered. After the round, every COP version, observer note and coaching log is lost from the evaluator's device. Only the plan and inject-delivery log remain. Behavior evidence cannot be recovered, and no step can be identified as performed or omitted by the team. There is no evidence that the team deleted information.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: NO

TC-S1 behavior evidence is insufficient because evaluator records were lost. Apply NO and document the cause and missing evidence, even though opportunity delivery is confirmed.

An inject plan does not establish team failure. Do not award 0 or reinterpret evaluator-side loss as team deletion of the COP. NO is excluded from scored denominators.

R10 · TC-K2 · Explain what an evidence item establishes

K / CPX-K · Open canonical criterion

Fictional rater boundary case. Document R10-S at 19:20 ICT, before the 19:30 round, declares a subset exercise in reading coordination-behavior evidence. It assigns no assessment-design or outcome–evidence explanation task and declares TC-K2 inapplicable to this scope. Scope-owner role O9 records the justification before the start. No full eight-criterion result is claimed and scope is not changed after seeing performance. This is a rater coding exercise, not a team-assessment round under the complete framework.

Before revealing: record your rating, evidence, rationale and lock time in your own form.

Reveal teaching key after locking ratings

Illustrative answer: NA

TC-K2 is outside the prespecified scope/role, with a justification owner, so NA applies. It is not retrospective coding caused by absent work.

NA is neither zero nor part of scored denominators. This subset must not be presented as a full eight-criterion team assessment. An accidentally omitted opportunity in a full round calls for considering NO, not retrospective scope revision.

Independent rating and review forms

Each file has 20 rows with matching task IDs and blank rating, reasoning and rater fields. Distribute A/B separately and lock originals before comparison. The review form separates original ratings from agreed ratings; leave agreed_score blank and record unresolved_dissent if unresolved.

Key fields: prior_exposure records task/key exposure; evidence_refs cites evidence; opportunity/prompting records opportunities and help; locked_at fixes the time before seeing others; a_original_ref/b_original_ref identify originals; resolution_reason explains the decision. Keep copies in authorized institutional storage and use codes instead of names.

Questions only (MD) · Key with rationales (MD) · JSON

Compare pre-discussion ratings

Enter only locked ratings after independent scoring is finished. Seeing A/B together on this screen is not independent assessment. Computation stays on this page, with no saving, submission or uploads; reloading clears the tool. Do not enter personal or incident data.

Manual check by set and criterion: eligible numeric pairs have both A/B in 0,1,2,3; exact matches have equal A/B; differences = eligible pairs minus matches. If reporting a proportion, use matches/eligible pairs and always display the denominator; zero denominator is undefined. Pairs involving NO/NA remain separate even when codes match. Blank means unentered, not NO or zero.

This tool checks A/B agreement only, not agreement with the key or correctness. Each criterion has very few examples. Use it to identify discussion points, not to certify reliability or readiness.

Without JavaScript, use the CSV forms and manual method above. Tasks and keys remain usable.

Return to the framework and learning pathways

Assessor guide · Team assessment · KSEC / OBE · Six case routes

Rationale sources and limits

Uses the framework/assessor guide's rules and design-rationale sources, retaining their access limits. Set design, tasks, keys and comparison tool are project syntheses, not a WHO or original-researcher training package. Ten items per set is not a reliability-study sample-size calculation.

กรมป้องกันและบรรเทาสาธารณภัย. แผนการป้องกันและบรรเทาสาธารณภัยแห่งชาติ พ.ศ. 2564–2570

Printed p.43, Figure 3–2 (PDF p.54); printed p.124, items 5.1–5.5 (PDF p.135).

Provides Thai mechanism context and information, option-development, coordination and operational-follow-up functions.

Inspected saved original-page images, not current local orders. Responsible officials must verify applicable plans and actual authority before exercise use.

Framework context