Question · NIST AI RMF · MEASURE
What ongoing-monitoring evidence does M-25-21 expect?
MEASURE uses quantitative, qualitative or mixed methods to analyze, benchmark and monitor AI risks, through 4 categories and 22 subcategories.
01 / The answer
The short answer
M-25-21 requires agencies to monitor high-impact AI on an ongoing basis, through testing and periodic human review where feasible, to find adverse effects on performance and security, including possible privacy, civil rights or civil liberties violations. Monitoring must be designed to detect unforeseen circumstances, changes to the system after deployment, and changes in the context of use or data. Agencies must apply mitigations, keep system and use documentation, and where possible make the evaluation traceable. Repeatable test results compared across runs are useful evidence. This lines up with AI RMF MEASURE 2.4, though the memo does not cite the RMF.
02 / The rule
What NIST AI RMF · MEASURE requires
MEASURE uses quantitative, qualitative or mixed methods to analyze, benchmark and monitor AI risks, through 4 categories and 22 subcategories. It is where documented test sets, metrics, deployment-like performance testing, production monitoring and evaluation of the measurement process itself are expected.
- 01Select metrics for the most significant mapped risks and regularly reassess metric appropriateness (MEASURE 1.1, 1.2).
- 02Document test sets, metrics and TEVV tools; measure performance under conditions similar to deployment (MEASURE 2.1, 2.3).
- 03Monitor functionality and behavior of the AI system in production (MEASURE 2.4).
- 04Demonstrate validity and reliability, document generalizability limits, and evaluate safety, security, explainability, privacy and fairness (MEASURE 2.5–2.11).
03 / Records
Records to keep
- 01TEVV plan with documented test sets, metrics and tools
- 02Evaluation reports under deployment-like conditions
- 03Production monitoring dashboards or logs with drift and incident records
- 04Review of measurement effectiveness and metric updates
Optional evidence · where Planisphere fits
Keep a record anyone can check
When your team meets a duty, Planisphere can seal the event it reports as a receipt.
A disclosure shown, a review signed, a model change approved: the event your system reports.
The event becomes a sealed receipt, dated and attributed to whoever is signed in.
Paste the receipt into Verify. No account or key.
A record supports your compliance evidence process. It is not a legal finding or a certification. Systems that call Planisphere's gate API can also hold or refuse an action before it runs.
SOURCES
Go back to the source
National Institute of Standards and Technology
National Institute of Standards and Technology
NIST Trustworthy and Responsible AI Resource Center
RELATED
Related: NIST AI RMF · MEASURE: the full guide · Which NIST AI RMF MEASURE subcategories can a third party evidence? · Does NIST AI RMF require independent / third-party evaluation?