Validator Queue
HUMAN REVIEW · STAGE 3
14
Awaiting Review
↑ 3 since yesterday
4.2
Avg Days in Queue
93%
AI Level Match Rate
↑ 2% this month
3
Escalated Cases
Submission Pipeline · Current Status
📥
Intake
Self-assessment
8
🤖
AI Pre-Scan
Signal extraction
5
👁️
Human Review
Signal verification
14
🤝
Peer Attestation
2 nominees
6
⚖️
Final Decision
Level assignment
3
✅
Published
Profile live
241
Cases Assigned to You
All
Discrepancy
L4–L5 Claims
Overdue
| Contributor | AI Suggested | Status | Priority | Age | |
|---|---|---|---|---|---|
DR Dr. Sofia Delgado CLIMATE RISK ECONOMICS |
L4 | Human Review | HIGH | 6 days | ··· |
MT Marcus Thibault ORGANIZATIONAL DESIGN |
L3 | Human Review | MED | 4 days | ··· |
AK Dr. Anika Khatri COMPUTATIONAL LINGUISTICS |
L5 | Dispute Flagged | HIGH | 9 days | ··· |
JO James Okafor VENTURE CAPITAL PRACTICE |
L2 | Human Review | LOW | 2 days | ··· |
LN Dr. Léa Nguyen BEHAVIORAL NEUROSCIENCE |
L4 | Peer Attestation | MED | 12 days | ··· |
Validator Workflow
COMPLETE REVIEW PROCESS · ALL 6 STAGES
📥
Stage 1 · Intake & Self-Assessment
Contributor completes the KWIS questionnaire. AI collects and flags their self-reported data for the pipeline.
- 1.1Contributor completes all 20 questionnaire questions across 6 sections, including self-assessed level (Q20) and explicit limits of expertise (Q18).
- 1.2System generates a unique KWIS submission ID and timestamps the entry. All answers are locked as submitted — no editing without a formal amendment request.
- 1.3Submission is routed to Stage 2 (AI Pre-Scan) automatically. Contributor receives confirmation email with their submission ID and expected timeline.
- 1.4If the contributor claims L4 or L5, the system flags the submission as HIGH priority and notifies the lead validator.
🤖
Stage 2 · AI Pre-Scan
Claude scans public sources to surface, corroborate, or challenge each self-reported credibility signal.
- 2.1AI searches for the contributor's name + domain across: Google Scholar, ORCID, ResearchGate, LinkedIn, institutional websites, conference archives, podcast databases, and major publication indexes.
- 2.2Each self-reported signal is tagged as one of: CORROBORATED (public source found), PARTIAL (related evidence found, not exact), UNVERIFIED (no public evidence), or CONFLICTED (public data contradicts claim).
- 2.3AI generates a preliminary level recommendation (L1–L5) based solely on verifiable public signals — not on self-reported claims.
- 2.4Discrepancy flag: If the AI's suggested level differs from the contributor's self-assessment by 2+ levels, the case is automatically escalated to HIGH priority for human review.
- 2.5AI pre-scan report is attached to the submission record and made available to the human validator in Stage 3.
AI Signal Status
Meaning
Validator Action
CORROBORATED
Public source confirms the claim
Quick-confirm or spot-check
PARTIAL
Related evidence, not exact match
Investigate and interpret
UNVERIFIED
No public evidence found
Request documentation from contributor
CONFLICTED
Public data contradicts claim
Flag for discrepancy review
👁️
Stage 3 · Human Review
The core validator judgment layer. A trained KWIS reviewer examines the AI pre-scan, verifies flagged signals, and forms an independent level assessment.
- 3.1Read the full submission first. Do not look at the AI suggested level until you have read the contributor's self-assessment in full. Form your own initial impression before consulting the AI report.
- 3.2Review the AI pre-scan signal by signal. For each CORROBORATED signal: confirm or note reservations. For each UNVERIFIED signal: determine if it requires documentation or can be assessed contextually. For each CONFLICTED signal: investigate the specific discrepancy.
- 3.3Pay particular attention to Question 18 (limits of expertise) and Question 19 (history of being wrong). Intellectual honesty in these answers is itself a credibility signal. Thin, defensive, or implausibly perfect answers to these questions should reduce the validator's confidence.
- 3.4Cross-reference the contributor's claimed contribution (Q10) against any available public record. If a specific outcome is claimed (e.g. "increased enrollment by 34%"), note whether this is verifiable or self-reported only.
- 3.5Record your independent level assessment before consulting the AI's suggestion. Then note any difference and the reason for your decision.
- 3.6If you require additional documentation from the contributor, use the "Request Documentation" function. This pauses the SLA clock. Document what you requested and why.
Mandatory Escalation to Lead Validator: L4 or L5 claims; CONFLICTED signals on core credentials; contributor/AI level difference of 2+; any indicator of credential fabrication.
Recommended Escalation: Domain is highly specialized (validator lacks sufficient context); contributor's field involves active public controversy; contributor has a significant existing public platform that could amplify a mis-assignment.
Flag for Peer Attestation: L3+ claims where the public record is sparse but the contribution description is detailed and plausible; first-time submissions from emerging practitioners.
🤝
Stage 4 · Peer Attestation
Two nominated peers from within the contributor's domain independently confirm or challenge the validator's provisional level assessment.
- 4.1Peer attestation is required for L3 and above, and optional (but encouraged) for L1–L2. Contributor nominates 2 peers; the system sends a structured, anonymized attestation form — peers do not see the contributor's self-assessment or AI report.
- 4.2Peers are asked to assess: (a) their knowledge of the contributor's work; (b) their assessment of the contributor's level; (c) any specific evidence they can point to; (d) any reservations they would record. Peer attestations are stored but not shown to the contributor.
- 4.3If both peers confirm the provisional level: proceed to Stage 5. If one peer disputes: validator reviews and documents reasoning. If both peers dispute or suggest a lower level: the case returns to Stage 3 for re-assessment.
- 4.4Peer attestation quality is tracked over time. Peers who consistently attest inflated levels, or who exclusively attest people they co-author with, are flagged for potential conflict-of-interest review.
⚖️
Stage 5 · Final Decision & Level Assignment
Lead validator issues the binding level assignment and notation. All supporting documentation is locked into the audit record.
- 5.1Lead validator reviews the full record: submission, AI pre-scan, human review notes, peer attestations. Issues final L1–L5 assignment with mandatory written rationale (minimum 100 words).
- 5.2The final record includes: assigned level, confidence tier (HIGH / MEDIUM / LOW), key supporting signals, any noted limitations, discrepancies, or caveats for the public-facing profile.
- 5.3Contributor is notified of their level. If assigned below their self-assessment, a brief, respectful rationale is provided. Contributor has 14 days to submit a formal appeal with new evidence. Appeals are reviewed by a second, independent validator.
- 5.4All decisions are final unless an appeal introduces materially new evidence. Level assignments carry a two-year validity window before perennial refresh is triggered.
✅
Stage 6 · Publication & Perennial Maintenance
Profile goes live, badge becomes portable and embeddable. The system monitors for major new signals and triggers re-assessment at the 2-year mark.
- 6.1Profile is published with KWIS ID, level badge, domain, key signals (summarized), confidence tier, and validity date. The full signal record is accessible to institutions and platforms via API.
- 6.2The AI monitoring system runs a quarterly scan for major new signals: new publications, significant speaking activity, new institutional affiliations, peer-nominated updates. If a signal threshold is crossed, contributor is invited to submit an amendment.
- 6.3At the 2-year mark, a lightweight perennial refresh is triggered — a 5-question update questionnaire focused on new activity. The full pipeline is only re-triggered if the level changes by 1 or more.
- 6.4Retroactive review: If a contributor is found to have made materially false claims after publication, their profile is immediately suspended pending investigation. A suspension notice replaces the badge during the review period.
Decision Framework
VALIDATOR JUDGMENT GUIDE
The Validator's Cardinal Rules
These rules govern every decision. They are not guidelines — they are the foundation of KWIS credibility.
- R1You are not granting a favor. You are issuing a public signal that readers will use to calibrate trust. Over-assigning levels is as harmful as under-assigning. Err toward honesty, not generosity.
- R2Domain specificity is non-negotiable. A contributor who is an eminent L4 in Macroeconomics may be an L1 in Health Economics. The assessment is domain-specific. Never let general eminence inflate a domain-specific score.
- R3Demonstrated beats claimed. A self-reported contribution without a traceable public record carries much less weight than a modest but verified publication. Your job is to find the evidence, not accept the narrative.
- R4Intellectual honesty is evidence. A contributor who names specific limits of their expertise, who has publicly corrected a past position, who hedges appropriately — this person is demonstrating a hallmark of genuine expertise. Weight it accordingly.
- R5If in doubt, go lower and explain clearly. It is always better to assign a lower level with a precise explanation than a higher level with vague support. The appeal process exists for contributors who deserve more. Inflation is irreversible.
Level Assignment Thresholds
Minimum signal requirements for each level assignment — all must be met, not just some.
Level
Minimum Verified Signals
Disqualifiers
L1 · Informed Observer
None required. General education or self-reported interest in the domain.
None — default level for insufficient signal.
L2 · Domain Student
Formal coursework, certification, or structured entry-level practice verified.
No verified formal engagement with the domain.
L3 · Practicing Expert
Advanced degree OR 5+ years practice PLUS at least one verifiable public output (publication, significant talk, recognized contribution).
Practice without public record. Degree without demonstrated practice.
L4 · Field Authority
All L3 criteria PLUS peer attestation PLUS evidence of domain influence (citations, editorial role, keynote history, advisory appointments).
Missing peer attestation. Influence limited to social media only.
L5 · Pioneer
All L4 criteria PLUS documented field-defining contribution (paradigm-shifting publication, transformative institutional legacy, or cross-domain foundational influence).
Influence not yet assessed as field-defining by the broader community.
Escalated Cases
REQUIRES LEAD VALIDATOR REVIEW
| Contributor | AI Suggested | Reason | Priority | Age | |
|---|---|---|---|---|---|
AK Dr. Anika Khatri COMPUTATIONAL LINGUISTICS |
L5 | Conflicted Signal | HIGH | 9 days | ··· |
DR Dr. Sofia Delgado CLIMATE RISK ECONOMICS |
L4 | L4 Claim | HIGH | 6 days | ··· |