AI Risk Assessment Tool

Comprehensive clinical AI system risk evaluation framework based on FDA Software as a Medical Device (SaMD) guidelines, ISO 14971:2019 risk management standards, and FMEA methodology.

⚠️

Critical Disclaimer: Educational Use Only

This risk assessment tool is for educational, research, and preliminary evaluation purposes only. It does NOT constitute: formal regulatory submission support, legal compliance certification, professional safety assessment, or official risk management documentation.Results must be reviewed by qualified professionals (regulatory affairs specialists, clinical safety officers, legal counsel) before making deployment decisions or regulatory submissions. This tool provides guidance based on established frameworks but cannot replace comprehensive risk management processes required by FDA, ISO 14971, or other regulatory bodies. Use of this tool does not ensure regulatory compliance or clinical safety. High-risk medical AI systems require expert multidisciplinary review.

Assessment Methodology

Framework Basis

This assessment integrates multiple evidence-based frameworks:

FDA SaMD: Risk categorization based on intended use, clinical significance, and state of healthcare situation
ISO 14971:2019: Medical device risk management including severity classification and risk control measures
FMEA: Failure Mode and Effects Analysis principles for systematic hazard identification
WHO AI Ethics: Principles for responsible AI in healthcare including fairness, transparency, and accountability

Scoring System

The assessment uses weighted scoring across 6 dimensions:

Clinical Impact (30%): Intended use, potential harm, clinical criticality, human oversight
Data Quality (25%): Representativeness, bias testing, completeness, labeling quality
Clinical Validation (25%): Validation rigor, comparative effectiveness, appropriate metrics, temporal testing
Transparency (10%): Explainability, documentation of limitations and failure modes
Regulatory (5%): Regulatory status, data protection compliance
Post-Market (5%): Surveillance, adverse event reporting, update management

Note: Higher scores indicate better risk management and safety controls. The weighted approach prioritizes clinical impact and validation evidence as most critical to patient safety.

Interpretation Guidelines

80-100% (Low Risk): Comprehensive risk management, ready for deployment with appropriate oversight
60-79% (Moderate Risk): Acceptable with identified gaps requiring mitigation before or shortly after deployment
40-59% (High Risk): Significant deficiencies requiring substantial additional work before deployment
0-39% (Critical Risk): Not suitable for clinical use; major development and validation needed

Each question includes methodology notes explaining the scientific basis for the assessment dimension.

Questions answered: 0 / 18

Clinical Impact & Purpose (Weight: 30%)

1. What is the primary intended use of the AI system?

Administrative/Non-clinicalScheduling, billing, resource management - no direct patient care impact

Clinical documentation supportAssists with note-taking, coding, summarization - indirect patient impact

Clinical decision support - informationalProvides information to inform clinical decisions without direct recommendation

Clinical decision support - diagnostic/therapeuticProvides specific diagnostic interpretations or treatment recommendations

Autonomous clinical actionMakes or executes clinical decisions with minimal human oversight (e.g., closed-loop insulin delivery)

Clinical Impact & Purpose (Weight: 30%)

2. What is the potential severity of harm if the AI system fails or provides incorrect output?

NegligibleInconvenience only, no patient safety impact, easily reversible

MinorTemporary discomfort, minimal intervention required, self-limiting

ModerateRequires medical intervention, potential for incorrect diagnosis or inappropriate treatment

SeriousPermanent impairment, major surgical intervention, life-altering consequences

Critical/CatastrophicLife-threatening condition or death

Clinical Impact & Purpose (Weight: 30%)

3. What is the clinical state or situation of concern?

Non-seriousStable condition, ample time for clinical review, low urgency

Serious but non-criticalSignificant condition but not immediately life-threatening, time for validation

Serious and urgentTime-sensitive decisions required (e.g., stroke, STEMI), limited time for review

Critical and time-sensitiveImmediately life-threatening, rapid intervention required (e.g., septic shock, cardiac arrest)

Critical with autonomous interventionAI controls life-sustaining therapy without real-time human oversight

Clinical Impact & Purpose (Weight: 30%)

4. How does the AI system integrate into the clinical workflow?

Human-in-the-loop (mandatory review)Clinician must review and approve every AI output before action

Human-on-the-loop (monitoring)Clinician monitors AI outputs with ability to intervene

Human-over-the-loop (periodic review)Clinician reviews AI performance periodically, not per case

Partial automation with optional reviewAI acts automatically but clinician can review if desired

Full automation (autonomous)AI operates independently without routine human review

Data Quality & Representativeness (Weight: 25%)

5. How representative is the training dataset of the intended patient population?

Highly representative & diverseMulti-site, diverse demographics (age, sex, race, ethnicity, socioeconomic), geographically varied

Mostly representativeMultiple sites, good demographic diversity, validated representativeness

Moderately representativeLimited sites, some demographic gaps identified and documented

Poorly representativeSingle site or narrow population, significant demographic gaps

Unknown or highly biasedNo diversity analysis, convenience sample, or known selection bias

Data Quality & Representativeness (Weight: 25%)

6. Has the AI model been explicitly tested for fairness and bias across demographic subgroups?

Comprehensive bias audit publishedDetailed subgroup analysis across multiple dimensions, peer-reviewed, mitigation strategies implemented

Rigorous internal bias testingSystematic testing across key demographics, results documented, some mitigation

Basic fairness assessmentLimited subgroup analysis, major groups tested, minimal mitigation

Minimal or ad-hoc testingInformal fairness checks, incomplete analysis

No bias assessmentNo fairness testing conducted or documented

Data Quality & Representativeness (Weight: 25%)

7. What is the quality and completeness of the training data?

ExcellentComprehensive quality assurance, <5% missing data, documented provenance, expert validation of labels

GoodStructured QA process, <15% missing data, mostly complete labels

FairBasic QA, 15-30% missing data, some labeling inconsistencies identified

Poor>30% missing data, significant labeling errors, limited QA

Unknown or very poorNo QA process, data completeness unknown, unvalidated labels

Data Quality & Representativeness (Weight: 25%)

8. How was data labeling/ground truth established?

Expert consensus with validationMultiple expert raters, formal consensus process (e.g., Delphi), inter-rater reliability measured

Multiple expert ratersAt least 2 independent experts, disagreements adjudicated

Single expertOne qualified clinician, systematic labeling protocol

Semi-automated or rule-basedAlgorithm-assisted labeling with some human review

Automated or unclearFully automated labels, crowdsourcing without expert validation, or unknown process

Clinical Validation & Performance (Weight: 25%)

9. What level of clinical validation has been completed?

Prospective multi-center RCTRandomized controlled trial across multiple sites with patient outcomes measured

Prospective multi-site cohortProspective validation at 3+ independent external sites

Prospective single-siteProspective validation at one site distinct from training data

Retrospective external validationRetrospective testing on external dataset not used in training

Internal validation onlyOnly tested on held-out subset of training dataset, or no formal validation

Clinical Validation & Performance (Weight: 25%)

10. How does AI performance compare to the current standard of care?

Statistically superior with clinical significanceSignificantly better than standard of care with meaningful clinical impact (e.g., NNT, effect size)

Non-inferior with statistical rigorProven non-inferior to standard of care in well-powered study

Similar performanceComparable to standard of care but benefits unclear

Inferior in some metricsWorse than standard of care in some clinically relevant metrics

Unvalidated or clearly inferiorNo comparison study or significantly worse than standard of care

Clinical Validation & Performance (Weight: 25%)

11. What metrics were used to evaluate performance, and are they clinically meaningful?

Comprehensive clinical metricsMultiple metrics including calibration, decision curve analysis, NNT/NNH, patient-centered outcomes

Standard discrimination + calibrationSensitivity, specificity, PPV, NPV, calibration plots at relevant thresholds

Discrimination metrics onlyAUC-ROC, accuracy, but lacking calibration or threshold analysis

Limited metricsOnly accuracy or AUC without clinical context

Inappropriate or unclear metricsMetrics not suited to clinical problem or not reported

Clinical Validation & Performance (Weight: 25%)

12. Have temporal validation and/or performance monitoring over time been conducted?

Continuous monitoring with alertsReal-time performance dashboards, automated alerts for degradation, regular recalibration

Regular temporal validationQuarterly or annual prospective validation on new data

Initial temporal splitValidated on future time period during development, but no ongoing monitoring

No temporal validationRandom train/test split without temporal consideration

No monitoring planNo plan for performance tracking post-deployment

Transparency & Explainability (Weight: 10%)

13. How interpretable and explainable is the AI model to clinicians?

Fully interpretable by designInherently interpretable model (e.g., sparse linear, decision tree, rule list) with documented logic

Post-hoc explanation methods validatedSHAP, LIME, or attention maps provided and validated for faithfulness

Feature importance availableGlobal feature importance provided, but no per-prediction explanations

Limited interpretabilityMinimal explanation, complex black-box model without interpretation tools

Complete black boxNo explanation provided, model internals unknown to users

Transparency & Explainability (Weight: 10%)

14. Is there comprehensive documentation of model limitations, failure modes, and contraindications?

Comprehensive user documentationDetailed limitations, known failure modes, contraindications, edge cases documented and communicated

Good documentationMajor limitations documented, some failure mode analysis provided

Basic documentationGeneral limitations mentioned, minimal failure mode detail

Minimal documentationVague or incomplete limitation statements

No limitations documentedNo discussion of limitations, failure modes, or contraindications

Regulatory & Compliance (Weight: 5%)

15. What is the regulatory status and pathway for this AI system?

FDA cleared/approved Class III (PMA)Premarket Approval, highest risk class, most rigorous review

FDA cleared Class II (510(k))Premarket notification demonstrating substantial equivalence

FDA Class I or enforcement discretionLow risk device or FDA exercises enforcement discretion

Non-device software under quality systemNot regulated as device but has QMS (e.g., some CDSS)

No regulatory submissionAdministrative tool, not subject to medical device regulation

Regulatory & Compliance (Weight: 5%)

16. Does the system comply with data protection regulations (HIPAA, GDPR)?

Full compliance with independent auditHIPAA/GDPR compliant, third-party audit conducted, certifications obtained

Compliance verified internallyInternal compliance assessment completed, policies and technical controls in place

Partial compliance, gaps identifiedSome compliance measures, known gaps documented with mitigation plan

Compliance unclearNo formal assessment, compliance status uncertain

Known non-complianceIdentified violations or significant compliance gaps

Post-Market Surveillance (Weight: 5%)

17. Is there a system for post-market surveillance and adverse event reporting?

Comprehensive surveillance systemAutomated event detection, structured reporting to FDA/regulatory bodies, root cause analysis protocols

Active monitoring with reportingRegular performance review, adverse event reporting mechanism, user feedback loop

Passive monitoringUser-reported incidents collected, but no active surveillance

Minimal trackingAd-hoc incident tracking, no formal surveillance

No surveillance systemNo plan for post-market monitoring or adverse event reporting

Post-Market Surveillance (Weight: 5%)

18. Is there a plan for model updates, retraining, and change management?

Formal change control with PCCPFDA Predetermined Change Control Plan or equivalent, validation protocol for updates, version control

Defined update processScheduled retraining, validation requirements documented, change log maintained

Ad-hoc updates plannedUpdates anticipated but process not fully defined

Unclear update processNo defined plan for model maintenance

Static model, no updatesModel locked, no plan for updates or retraining

Please ensure all 18 questions are answered. Your responses will be used to calculate a weighted risk score and generate tailored recommendations.