Marketplace · Text & chat

Qwen2.5-7B-Instruct-Q4_K_M

Call Qwen2.5-7B-Instruct-Q4_K_M through one OpenAI-compatible API, served by independent machines at prices their owners set. There's no host for it on the grid at this moment (last live 15 Nov 2026) — so the next machine that loads it serves every request for it.

Hosts now
0
open slot
Size
4.4 GB
of weights
Last listed from
£0.03
per 1M output tokens

The grid is one machine short.

Qwen2.5-7B-Instruct-Q4_K_M is about 4.4 GB of weights, so a machine with a little more than that in free memory — a gaming GPU, an Apple-silicon Mac, a well-specced mini PC — can usually serve it. Set your own price — you keep 85% of what it earns, and new hosts start with 12 months fee-free (up to £250 of earnings). A payout fee of £2 + 1% per payout to a UK bank account, £2 + 2% to any other country applies when you're paid.

  1. Install the connector. One command on the machine that runs your models. It dials out over an encrypted link — no ports to open. The connector
  2. Load Qwen2.5-7B-Instruct-Q4_K_M. It has run here on llama_cpp; any supported engine works. The platform benchmarks it on your hardware first.
  3. Set your price. The last host listed it from £0.03 per 1M output tokens. Developers see your price beside the speed your machine measures.
  4. Get paid when it's called. Switch it off whenever you like. How hosting works

About Qwen2.5-7B-Instruct-Q4_K_M.

  • Type: Text & chat
  • Size: 7B parameters, about 4.4 GB of weights
  • Context window: up to 32,768 tokens
  • Runs on: llama_cpp
  • Last listed from: £0.03 per 1M output tokens
  • First on the grid: 8 Nov 2026
  • Hosts serving it now: 0
python · openai sdk
from openai import OpenAI
client = OpenAI(base_url="https://inpherio.co.uk/v1",
                api_key=INPHERIO_API_KEY)
reply = client.chat.completions.create(
    model="Qwen2.5-7B-Instruct-Q4_K_M",
    messages=[{"role": "user", "content": "Hello"}],
)

Qwen2.5-7B-Instruct-Q4_K_M in detail.

Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:

Published by
Qwen
Parameters
7.6B
Format
transformers
Architecture
qwen2
Licence
Apache 2.0
Based on
Qwen/Qwen2.5-7B
Task
text generation
Library
transformers
Languages
en
First published
16 Sep 2024
Last updated
12 Jan 2025
Downloads (30 days)
6 million on Hugging Face
Likes
2,487

qwen2 text-generation chat en

The model card

Published by the model's authors on Hugging Face — shown as they wrote it, unedited by Inpherio.

Qwen2.5-7B-Instruct

Introduction

Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:

  • Significantly more knowledge and has greatly improved capabilities in coding and mathematics, thanks to our specialized expert models in these domains.
  • Significant improvements in instruction following, generating long texts (over 8K tokens), understanding structured data (e.g, tables), and generating structured outputs especially JSON. More resilient to the diversity of system prompts, enhancing role-play implementation and condition-setting for chatbots.
  • Long-context Support up to 128K tokens and can generate up to 8K tokens.
  • Multilingual support for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.

This repo contains the instruction-tuned 7B Qwen2.5 model, which has the following features:

  • Type: Causal Language Models
  • Training Stage: Pretraining & Post-training
  • Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
  • Number of Parameters: 7.61B
  • Number of Paramaters (Non-Embedding): 6.53B
  • Number of Layers: 28
  • Number of Attention Heads (GQA): 28 for Q and 4 for KV
  • Context Length: Full 131,072 tokens and generation 8192 tokens
  • Please refer to this section for detailed instructions on how to deploy Qwen2.5 for handling long texts.

For more details, please refer to our blog, GitHub, and Documentation.

Requirements

The code of Qwen2.5 has been in the latest Hugging face transformers and we advise you to use the latest version of transformers.

With transformers<4.37.0, you will encounter the following error:

KeyError: 'qwen2'

Quickstart

Here provides a code snippet with apply_chat_template to show you how to load the tokenizer and model and how to generate contents.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen2.5-7B-Instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Processing Long Texts

The current config.json is set for context length up to 32,768 tokens. To handle extensive inputs exceeding 32,768 tokens, we utilize YaRN, a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.

For supported frameworks, you could add the following to config.json to enable YaRN:

{
  ...,
  "rope_scaling": {
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "type": "yarn"
  }
}

For deployment, we recommend using vLLM. Please refer to our Documentation for usage if you are not familar with vLLM. Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required.

Read the rest of the model card

Evaluation & Performance

Detailed evaluation results are reported in this 📑 blog.

For requirements on GPU memory and the respective throughput, see results here.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{qwen2.5,
    title = {Qwen2.5: A Party of Foundation Models},
    url = {https://qwenlm.github.io/blog/qwen2.5/},
    author = {Qwen Team},
    month = {September},
    year = {2024}
}

@article{qwen2,
      title={Qwen2 Technical Report}, 
      author={An Yang and Baosong Yang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Zhou and Chengpeng Li and Chengyuan Li and Dayiheng Liu and Fei Huang and Guanting Dong and Haoran Wei and Huan Lin and Jialong Tang and Jialin Wang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Ma and Jin Xu and Jingren Zhou and Jinze Bai and Jinzheng He and Junyang Lin and Kai Dang and Keming Lu and Keqin Chen and Kexin Yang and Mei Li and Mingfeng Xue and Na Ni and Pei Zhang and Peng Wang and Ru Peng and Rui Men and Ruize Gao and Runji Lin and Shijie Wang and Shuai Bai and Sinan Tan and Tianhang Zhu and Tianhao Li and Tianyu Liu and Wenbin Ge and Xiaodong Deng and Xiaohuan Zhou and Xingzhang Ren and Xinyu Zhang and Xipin Wei and Xuancheng Ren and Yang Fan and Yang Yao and Yichang Zhang and Yu Wan and Yunfei Chu and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhihao Fan},
      journal={arXiv preprint arXiv:2407.10671},
      year={2024}
}

Sources: Qwen/Qwen2.5-7B-Instruct on Hugging Face. Collected from public sources on 8 Nov 2026; the host serving it may run a different build or quantisation — the marketplace shows each host's own measurements.

Live now instead.

Similar text & chat models with hosts serving them right now.

Every live model →

Questions about Qwen2.5-7B-Instruct-Q4_K_M.

Can I use Qwen2.5-7B-Instruct-Q4_K_M through an API?

Yes. Inpherio serves Qwen2.5-7B-Instruct-Q4_K_M through one OpenAI-compatible API whenever a host on the grid is running it — the code on this page works unchanged. With no host right now, a call returns a "not available on any node" error and is never charged, and similar models are live on the marketplace.

How do I host Qwen2.5-7B-Instruct-Q4_K_M and get paid?

Install the Inpherio connector on the machine that runs your models, load Qwen2.5-7B-Instruct-Q4_K_M in a supported engine, and set a price. The platform benchmarks it on your hardware, then requests for it reach your machine. Set your own price — you keep 85% of what it earns, and new hosts start with 12 months fee-free (up to £250 of earnings). A payout fee of £2 + 1% per payout to a UK bank account, £2 + 2% to any other country applies when you're paid.

What hardware does Qwen2.5-7B-Instruct-Q4_K_M need?

Its weights are about 4.4 GB, so plan for a little more than that in free GPU or unified memory. Many gaming GPUs, Apple-silicon Macs and mini PCs qualify.

Why is Qwen2.5-7B-Instruct-Q4_K_M not being served right now?

Hosts on Inpherio are independent machines that come and go. It was last live on 15 Nov 2026. When a host loads it again, this page shows live prices and measured speeds.