Document value grounding · benchmark
Is a claim actually supported by the document it cites? Models are ranked by ROC-AUC on the open grounding-benchmark — with the headline metric being number grounding, where general NLI models fail.
Highlighted rows are Nutrient models. open = downloadable weights; closed = commercial (contact us). Ties broken by number AUC. Ctx = the model's context window in tokens — models with a shorter window truncate long premises (tables/filings), a real disadvantage reflected in the scores. Speed = inference throughput (pairs/s) on the standard hardware, a single NVIDIA RTX A6000 (batch 64, max len 1024).
Scoring is self-service and reproducible — no server in the loop. Score any NLI cross-encoder and open a PR with the results JSON.
pip install torch transformers datasets python benchmark/score.py --model <your-model> --config en --name "Your Model" --open true
Full instructions & rules: SUBMISSION.md.
The reference scorer score.py lives alongside the dataset.
Models benchmarked: nutrientdocs/grounding-en · nutrientdocs/grounding-multilingual · cross-encoder/nli-deberta-v3-base · cross-encoder/nli-MiniLM2-L6-H768 · facebook/bart-large-mnli · FacebookAI/roberta-large-mnli · joeddav/xlm-roberta-large-xnli · MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli · MoritzLaurer/deberta-v3-large-zeroshot-v2.0 · MoritzLaurer/mDeBERTa-v3-base-mnli-xnli · MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7