Marketplace · Text & chat

bespoke-minicheck:7b

Call bespoke-minicheck:7b through one OpenAI-compatible API. It's served by 1 host on the Inpherio grid, each at a price it sets — your request goes to the cheapest healthy one, and fails over to the next if it's down.

From
£0.06
per 1M output tokens
£1 buys about 17 million tokens (≈ 13 million words)
Hosts
1
Loaded now
0

At a glance.

  • Type: Text & chat
  • Hosts serving it now: 1 (0 with it loaded in memory)
  • Engines: ollama
  • Context window: up to 32,768 tokens
  • Rentable: a whole host from £0.60 an hour (rentals)
python · openai sdk
from openai import OpenAI
client = OpenAI(base_url="https://inpherio.co.uk/v1",
                api_key=INPHERIO_API_KEY)
reply = client.chat.completions.create(
    model="bespoke-minicheck:7b",
    messages=[{"role": "user", "content": "Hello"}],
)

bespoke-minicheck:7b in detail.

A state-of-the-art fact-checking model developed by Bespoke Labs.

Published by
bespokelabs
Parameters
7.7B
Download size
4.7 GB (as packaged for Ollama)
Quantisation
Q4_K_M
Format
gguf
Architecture
internlm2
Task
text classification
Languages
en
First published
9 Aug 2024
Last updated
20 Dec 2024
Downloads (30 days)
2,212 on Hugging Face
Likes
94

internlm2 text-classification en

The model card

Published by the model's authors on Hugging Face — shown as they wrote it, unedited by Inpherio.

Llama-3.1-Bespoke-MiniCheck-7B

This is a fact-checking model developed by Bespoke Labs and maintained by Liyan Tang and Bespoke Labs. The model is an improvement of the MiniCheck model proposed in the following paper:

📃 MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents, EMNLP 2024

GitHub Repo

The model takes as input a document and a sentence and determines whether the sentence is supported by the document: MiniCheck-Model(document, claim) -> {0, 1}

In order to fact-check a multi-sentence claim, the claim should first be broken up into sentences. The document does not need to be chunked unless it exceeds 32K tokens. Depending on use cases, adjusting chunk size may yield better performance.

Llama-3.1-Bespoke-MiniCheck-7B is finetuned from internlm/internlm2_5-7b-chat (Cai et al., 2024) on the combination of 35K data points only:

  • 21K ANLI examples (Nie et al., 2020)
  • 14K synthetically-generated examples following the scheme in the MiniCheck paper, but with additional proprietary data curation techniques (sampling, selecting additional high quality data sources, etc.) from Bespoke Labs. Specifically, we generate 7K "claim-to-document" (C2D) and 7K "doc-to-claim" (D2C) examples. The following steps were taken to avoid benchmark contamination: the error types of the model in the benchmark data were not used, and the data sources were curated independent of the benchmark.

All synthetic data is generated by meta-llama/Meta-Llama-3.1-405B-Instruct, thus the name Llama-3.1-Bespoke-MiniCheck-7B.

While scaling up the model (compared to what is in MiniCheck) helped, many improvements come from high-quality curation, thus establishing the superiority of Bespoke Labs's curation technology.

Model Variants

We also have other three MiniCheck model variants:

Model Performance

The performance of these models is evaluated on our new collected benchmark (unseen by our models during training), LLM-AggreFact, from 11 recent human annotated datasets on fact-checking and grounding LLM generations. Llama-3.1-Bespoke-MiniCheck-7B is the SOTA fact-checking model despite its small size.

Model Usage

Please run the following command to install the MiniCheck package and all necessary dependencies.

pip install "minicheck[llm] @ git+https://github.com/Liyan06/MiniCheck.git@main"

Below is a simple use case

from minicheck.minicheck import MiniCheck
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

doc = "A group of students gather in the school library to study for their upcoming final exams."
claim_1 = "The students are preparing for an examination."
claim_2 = "The students are on vacation."

### model_name can be one of:
### ['roberta-large', 'deberta-v3-large', 'flan-t5-large', 'Bespoke-MiniCheck-7B']
scorer = MiniCheck(model_name='Bespoke-MiniCheck-7B', enable_prefix_caching=False, cache_dir='./ckpts')
pred_label, raw_prob, _, _ = scorer.score(docs=[doc, doc], claims=[claim_1, claim_2]) # can set `chunk_size=your-specified-value` here, default to 32K chunk size. 

print(pred_label) # [1, 0]
print(raw_prob)   # [0.9840446675150499, 0.010986349594852094]
Read the rest of the model card

Throughput

We speed up Llama-3.1-Bespoke-MiniCheck-7B inference with vLLM. Based on our test on a single A6000 (48 VRAM), Llama-3.1-Bespoke-MiniCheck-7B with vLLM and MiniCheck-Flan-T5-Large have throughputs > 500 docs/min.

Automatic Prefix Caching

Automatic Prefix Caching (APC in short) caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part.

To enable automatic prefix caching for Bespoke-MiniCheck-7B, simply set enable_prefix_caching=True when initializing the MiniCheck model (no other changes are needed):

scorer = MiniCheck(model_name='Bespoke-MiniCheck-7B', enable_prefix_caching=True, cache_dir='./ckpts')

How automatic prefix caching affects the throughput and model performance can be found in the GitHub Repo.

Test on our LLM-AggreFact Benchmark

import pandas as pd
from datasets import load_dataset
from minicheck.minicheck import MiniCheck
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

### load 29K test data
df = pd.DataFrame(load_dataset("lytang/LLM-AggreFact")['test'])
docs = df.doc.values
claims = df.claim.values

scorer = MiniCheck(model_name='Bespoke-MiniCheck-7B', enable_prefix_caching=False, cache_dir='./ckpts')
pred_label, raw_prob, _, _ = scorer.score(docs=docs, claims=claims)  # ~ 500 docs/min, depending on hardware

To evaluate the result on the benchmark

from sklearn.metrics import balanced_accuracy_score

df['preds'] = pred_label
result_df = pd.DataFrame(columns=['Dataset', 'BAcc'])
for dataset in df.dataset.unique():
    sub_df = df[df.dataset == dataset]
    bacc = balanced_accuracy_score(sub_df.label, sub_df.preds) * 100
    result_df.loc[len(result_df)] = [dataset, bacc]

result_df.loc[len(result_df)] = ['Average', result_df.BAcc.mean()]
result_df.round(1)

License

This work is licensed under CC BY-NC 4.0. For commercial licensing, please contact [email protected].

Citation

@InProceedings{tang-etal-2024-minicheck,
  title = {MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents},
  author = {Liyan Tang and Philippe Laban and Greg Durrett},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  publisher = {Association for Computational Linguistics},
  url = {https://arxiv.org/pdf/2404.10774}
}

@misc{tang2024bespokeminicheck,
  title={Bespoke-Minicheck-7B},
  author={Bespoke Labs},
  year={2024},
  url={https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B},
}

Acknowledgements

Model perfected at Bespoke Labs.

Team:

  1. Liyan Tang
  2. Negin Raoof
  3. Trung Vu
  4. Greg Durrett
  5. Alex Dimakis
  6. Mahesh Sathiamoorthy

We also thank Giannis Daras for feedback and Sarthak Malhotra for market research.

Sources: bespokelabs/Bespoke-MiniCheck-7B on Hugging Face · bespoke-minicheck:7b on Ollama. Collected from public sources on 14 Nov 2026; the host serving it may run a different build or quantisation — the marketplace shows each host's own measurements.

Who's serving it.

Every host benchmarked this model on its own hardware before serving it. Prices are set by each host; health is scored by the platform from what it observes.

Hosts serving bespoke-minicheck:7b right now
HostTierPriceMeasured speedHealthStateRent
GTX 1080 Ti (11G)L1£0.03 in · £0.06 out7 tok/s · 181 ms first token78loads on request£0.60/h

Live data, refreshed on every visit. How billing works.

Room on the grid for your machine.

Right now one machine serves bespoke-minicheck:7b. If it drops, so does every request for it — a second host is what turns it into failover, and the grid needs that more than another copy. The router weighs price, measured speed, health and whether the model is already loaded — so a host with a sharper price or quicker hardware wins the next call.

Hosts now
1
0 loaded in memory
Price to beat
£0.06
per 1M output tokens
Fastest measured
7 tok/s

Set your own price — you keep 85% of what it earns, and new hosts start with 12 months fee-free (up to £250 of earnings). A payout fee of £2 + 1% per payout to a UK bank account, £2 + 2% to any other country applies when you're paid. Install the connector, load bespoke-minicheck:7b, and the platform benchmarks it on your hardware before your first request.