Add human-centered trust & uncertainty metrics to Hugging Face Evaluate

README.md

@@ -82,3 +82,4 @@ See this [step-by-step guide](https://huggingface.co/docs/evaluate/creating_and_
 ## Credits
 
 Thanks to [@marella](https://github.com/marella) for letting us use the `evaluate` namespace on PyPi previously used by his [library](https://github.com/marella/evaluate).
+

data/human_ai_trust_demo.csv

@@ -0,0 +1,121 @@
+prediction,reference,confidence,human_trust,belief_prior,belief_posterior,explanation_length
+0,0,0.462,0.389,0.462,0.462,111
+1,1,0.746,0.768,0.695,0.721,57
+0,0,0.759,0.764,0.572,0.665,63
+0,0,0.99,0.925,0.368,0.679,64
+0,0,0.721,0.936,0.396,0.559,99
+1,1,0.795,0.859,0.199,0.497,72
+0,0,0.745,0.542,0.396,0.571,86
+0,0,0.575,0.593,0.342,0.458,104
+0,0,0.921,0.855,0.465,0.693,90
+0,1,0.69,0.775,0.235,0.462,98
+0,0,0.869,0.789,0.504,0.686,64
+0,0,0.614,0.602,0.707,0.66,86
+0,0,0.96,0.99,0.378,0.669,38
+0,0,0.54,0.626,0.48,0.51,107
+1,1,0.838,0.718,0.538,0.688,61
+1,0,0.792,0.759,0.32,0.556,60
+0,1,0.481,0.434,0.445,0.463,105
+0,1,0.532,0.467,0.403,0.467,95
+0,1,0.612,0.789,0.42,0.516,113
+0,0,0.674,0.715,0.245,0.46,97
+0,1,0.414,0.288,0.405,0.409,120
+0,0,0.76,0.852,0.5,0.63,93
+1,1,0.591,0.803,0.69,0.64,99
+1,1,0.821,0.924,0.592,0.706,62
+1,1,0.612,0.46,0.831,0.721,111
+1,1,0.982,0.934,0.247,0.615,57
+1,1,0.633,0.759,0.574,0.603,83
+1,1,0.702,0.631,0.437,0.569,83
+1,1,0.872,0.916,0.838,0.855,56
+1,1,0.565,0.643,0.238,0.402,88
+1,0,0.627,0.535,0.232,0.43,103
+0,0,0.946,0.94,0.28,0.613,75
+1,1,0.509,0.185,0.05,0.279,94
+1,1,0.778,0.675,0.295,0.536,82
+1,1,0.789,0.764,0.248,0.519,68
+0,0,0.867,0.742,0.43,0.649,61
+1,1,0.564,0.728,0.468,0.516,96
+1,0,0.442,0.299,0.775,0.608,108
+0,0,0.828,0.784,0.59,0.709,71
+0,0,0.795,0.808,0.285,0.54,66
+0,0,0.788,0.932,0.22,0.504,78
+0,0,0.802,0.658,0.498,0.65,73
+1,1,0.648,0.764,0.136,0.392,89
+1,1,0.785,0.786,0.766,0.776,66
+1,1,0.794,0.696,0.636,0.715,68
+0,1,0.514,0.56,0.306,0.41,115
+1,1,0.99,0.99,0.057,0.524,56
+0,0,0.821,0.761,0.671,0.746,61
+1,1,0.571,0.578,0.377,0.474,102
+0,1,0.679,0.64,0.648,0.663,96
+0,0,0.604,0.615,0.081,0.342,80
+1,1,0.868,0.934,0.28,0.574,71
+0,0,0.924,0.99,0.401,0.662,52
+1,1,0.627,0.503,0.409,0.518,100
+1,0,0.716,0.929,0.31,0.513,76
+0,1,0.65,0.454,0.525,0.587,74
+0,1,0.699,0.683,0.186,0.443,69
+0,0,0.99,0.99,0.372,0.681,51
+1,0,0.571,0.599,0.424,0.497,104
+0,0,0.637,0.575,0.503,0.57,84
+1,0,0.493,0.472,0.542,0.518,117
+0,0,0.628,0.578,0.175,0.401,78
+0,0,0.738,0.679,0.093,0.416,80
+0,0,0.801,0.886,0.656,0.728,61
+0,0,0.792,0.827,0.466,0.629,68
+0,1,0.699,0.63,0.25,0.475,86
+1,1,0.752,0.842,0.71,0.731,71
+0,0,0.968,0.99,0.423,0.696,49
+0,1,0.568,0.65,0.636,0.602,111
+1,1,0.99,0.99,0.414,0.702,45
+1,1,0.844,0.761,0.812,0.828,77
+1,1,0.621,0.565,0.751,0.686,84
+0,0,0.589,0.664,0.35,0.47,104
+0,1,0.658,0.719,0.594,0.626,105
+1,0,0.573,0.571,0.529,0.551,76
+1,1,0.857,0.869,0.674,0.765,59
+1,1,0.821,0.949,0.207,0.514,77
+0,1,0.591,0.532,0.537,0.564,97
+0,0,0.623,0.678,0.612,0.617,98
+1,1,0.523,0.503,0.05,0.286,101
+0,0,0.683,0.661,0.163,0.423,88
+1,1,0.878,0.988,0.05,0.464,63
+0,0,0.782,0.865,0.346,0.564,87
+0,1,0.451,0.532,0.544,0.497,118
+1,0,0.621,0.751,0.7,0.661,98
+1,0,0.646,0.648,0.415,0.531,88
+1,1,0.617,0.686,0.726,0.672,91
+0,0,0.773,0.742,0.124,0.449,72
+1,1,0.759,0.791,0.059,0.409,82
+1,1,0.579,0.566,0.389,0.484,96
+1,1,0.804,0.813,0.477,0.64,76
+1,1,0.834,0.894,0.393,0.614,90
+1,1,0.912,0.831,0.05,0.481,69
+1,1,0.908,0.99,0.382,0.645,57
+0,1,0.435,0.334,0.139,0.287,129
+1,1,0.609,0.488,0.534,0.572,92
+1,1,0.827,0.943,0.473,0.65,50
+1,1,0.827,0.906,0.212,0.52,60
+1,1,0.827,0.89,0.297,0.562,52
+0,0,0.99,0.99,0.188,0.589,47
+0,0,0.836,0.834,0.387,0.612,69
+1,1,0.92,0.831,0.591,0.756,76
+1,1,0.893,0.901,0.203,0.548,66
+0,1,0.678,0.61,0.501,0.589,86
+1,1,0.703,0.8,0.294,0.498,93
+1,1,0.864,0.849,0.241,0.553,44
+1,1,0.634,0.552,0.379,0.506,96
+1,1,0.714,0.682,0.193,0.454,91
+1,1,0.677,0.718,0.289,0.483,73
+0,0,0.762,0.706,0.16,0.461,89
+1,1,0.99,0.908,0.793,0.891,54
+0,0,0.47,0.494,0.407,0.438,109
+1,1,0.853,0.877,0.26,0.556,73
+0,1,0.406,0.356,0.443,0.425,143
+1,0,0.543,0.496,0.378,0.46,106
+0,1,0.731,0.754,0.356,0.543,84
+0,0,0.76,0.615,0.523,0.641,74
+0,1,0.471,0.33,0.552,0.511,105
+1,1,0.643,0.571,0.294,0.468,101
+1,0,0.682,0.66,0.285,0.483,79

data/validate.py

@@ -0,0 +1,19 @@
+import evaluate
+import pandas as pd
+
+metric = evaluate.load("human_ai_trust")
+
+df = pd.read_csv("data/human_ai_trust_demo.csv")
+
+out = metric.compute(
+    predictions=df["prediction"].tolist(),
+    references=df["reference"].tolist(),
+    confidences=df["confidence"].tolist(),
+    human_trust_scores=df["human_trust"].tolist(),
+    belief_priors=df["belief_prior"].tolist(),
+    belief_posteriors=df["belief_posterior"].tolist(),
+    explanation_complexity=df["explanation_length"].tolist(),
+)
+
+print(out)
+

metrics/human_ai_trust/README.md

@@ -0,0 +1,133 @@
+# Human Trust & Uncertainty Metrics
+
+This module provides a human-centered evaluation framework for AI systems that operationalizes:
+
+- trust calibration  
+- belief updating  
+- uncertainty alignment  
+- asymmetric harm from overconfident errors  
+- explanation–confidence coupling  
+
+It complements traditional performance metrics by surfacing how users interpret, trust, and act on model outputs under uncertainty.
+
+**Trust calibration** refers to the alignment between a user's level of trust in an AI system and the system's actual reliability or confidence. Well-calibrated trust means users trust the AI appropriately—more when it's confident and correct, less when it's uncertain or error-prone.
+
+---
+
+## Why These Metrics Exist
+
+Accuracy alone is insufficient for evaluating AI systems in high-stakes or vulnerable contexts.
+
+Human decision-makers experience AI failures asymmetrically:
+a confident but wrong prediction is far more damaging than a hesitant error.
+
+Moreover, highly accurate systems can still cause harm if they:
+
+- induce over-reliance (automation bias)  
+- induce under-reliance (algorithmic aversion)  
+- miscommunicate uncertainty  
+- distort user beliefs  
+
+This metric suite provides theory-grounded, computational signals for evaluating these human-centered failure modes.
+
+---
+
+## Metrics Included
+
+| Metric | What It Measures |
+|--------|------------------|
+| Expected Trust Error (ETE) | Misalignment between human trust and model confidence |
+| Trust Sensitivity Index (TSI) | Responsiveness of trust to uncertainty signals |
+| Belief Shift Magnitude (BSM) | Degree of belief updating after AI exposure |
+| Overconfidence Penalty (OCP) | Asymmetric harm from confident but wrong predictions |
+| OCP (normalized) | Scale-invariant version of OCP |
+| Explanation–Confidence Alignment (ECA) | Coupling between explanation form and model confidence |
+
+---
+
+## Usage
+
+```python
+import evaluate
+
+metric = evaluate.load("human_ai_trust")
+
+out = metric.compute(
+    predictions=[1, 0, 1],
+    references=[1, 1, 0],
+    confidences=[0.9, 0.7, 0.8],
+    human_trust_scores=[0.85, 0.6, 0.75],
+    belief_priors=[0.3, 0.4, 0.5],
+    belief_posteriors=[0.6, 0.5, 0.7],
+    explanation_complexity=[10, 20, 15],
+)
+
+print(out)
+```
+
+---
+
+## Interpretation Guide
+
+**Low ETE + High TSI**  
+→ well-calibrated, uncertainty-sensitive users
+
+**High ETE + High TSI**  
+→ sensitive but miscalibrated trust
+
+**Low TSI**  
+→ users ignore uncertainty signals
+
+**High OCP**  
+→ confident errors dominate harm
+
+**High BSM**  
+→ strong AI influence on beliefs
+
+**Strong ECA (±)**  
+→ explanation style tracks uncertainty
+
+---
+
+## Limitations
+
+- These are descriptive metrics, not causal estimators
+- They do not evaluate explanation correctness or faithfulness
+- They do not measure fairness or bias
+- They assume confidence and trust are provided on comparable scales
+- Synthetic datasets cannot substitute for real human data
+
+---
+
+## Theory Foundations
+
+These metrics draw on:
+
+- calibration in probabilistic judgment
+- trust in automation
+- Bayesian belief updating
+- overconfidence bias
+- explanation trust mismatch
+
+They operationalize core human-centered constructs into computational form.
+
+---
+
+## Companion Dataset
+
+A small reference dataset is available at:
+
+**dyra-12/human_ai_trust_demo**
+
+It demonstrates:
+
+- trust calibration
+- belief updating
+- uncertainty communication
+- explanation–confidence alignment
+
+---
+
+## License
+
+Apache 2.0

metrics/human_ai_trust/human_ai_trust.json

@@ -0,0 +1,16 @@
+{
+  "description": "Human-centered evaluation metrics for trust calibration, belief updating, and uncertainty alignment in AI systems.",
+  "citation": "@misc{human_ai_trust2026,\n  title={Human Trust & Uncertainty Metrics for AI Evaluation},\n  author={Dyra},\n  year={2026},\n  note={Hugging Face Evaluate metric}\n}",
+  "homepage": "https://huggingface.co/datasets/Dyra1204/human_ai_trust_demo",
+  "license": "Apache-2.0",
+  "inputs_description": "predictions: List[Any], references: List[Any], confidences: List[float], human_trust_scores: List[float], belief_priors: Optional[List[float]], belief_posteriors: Optional[List[float]], explanation_complexity: Optional[List[float]]",
+  "keywords": [
+    "trust",
+    "uncertainty",
+    "human-ai-interaction",
+    "calibration",
+    "interpretability",
+    "belief-updating",
+    "hcai"
+  ]
+}