Learn more：https://treerouter.com
A model can answer a question convincingly and still fail inside your application. It may return the wrong JSON types, select an incorrect tool, overlook a business rule, or consume more requests than expected.
When evaluating Chinese LLMs such as DeepSeek, Qwen, Kimi, and GLM, begin with the exact models and endpoints you plan to use. Family names alone are not a testable configuration.
This guide provides five acceptance tests for an API evaluation. The examples are proposed fixtures, not reported model results. Run them against your own candidates and keep the outputs.
1. Freeze the comparison setup
Record the provider, model identifier, API protocol, client version, reasoning settings, and output limits. Use the same input files and acceptance criteria across candidates.
Do not force unsupported parameters into a supposedly identical request. Instead, record any differences required by each API.
DeepSeek’s official documentation, for example, describes OpenAI- and Anthropic-compatible access. That establishes documented access formats for that service; it does not establish identical support across every provider or every feature. DeepSeek API Docs
Start with a small representative sample. A useful initial set can contain extraction tasks, bilingual summaries, and tool-assisted lookups.
2. Test JSON syntax and business meaning separately
Use this fixture:
Order: A-104
Quantity: 3
Unit price: USD 12.50
Delivery date: not provided
Return JSON with:
order_id: string
quantity: integer
unit_price: number
currency: string
delivery_date: string or null
Do not invent missing information.Check whether the response parses. Then check the fields independently: quantity should be an integer, unit_price should be numeric, and the missing delivery date should remain null.
Valid JSON is not the same as correct data.
When using DeepSeek’s documented JSON Output mode, its guide specifies response_format={"type":"json_object"}, an instruction mentioning JSON, and sufficient output allowance. Those are provider-specific instructions; use the relevant documentation for other candidates. DeepSeek API Docs
3. Test a complete tool cycle
Define a harmless lookup tool such as lookup_order_status. Give it a fixed test record and ask:
> What is the status of order A-104?
Evaluate the whole sequence:
- The model requests the correct tool.
- The arguments contain the correct order identifier.
- Your application executes the lookup.
- The tool result is returned to the model.
- The final answer matches the result.
The application, not the model’s text response, performs the actual function execution. DeepSeek’s tool-calling documentation explicitly separates the model’s requested function from the client-provided execution. DeepSeek API Docs
Add a second case where the order identifier is missing. The desired behavior is to request clarification rather than invent an identifier.
For an agent application, passing this cycle matters more than simply producing fluent chat.
4. Test bilingual accuracy and conflicting context
Prepare a short Chinese product brief and request an English summary. Include details that are easy to distort:
- A product code that must remain unchanged.
- A number with a unit.
- A limitation that must not disappear.
- An uncertain launch date.
- A feature explicitly described as unavailable.
Review factual preservation separately from writing quality. A smooth English paragraph can still turn “planned” into “available.”
Next, provide two dated policy excerpts that disagree. Ask the model to identify the conflict and apply an explicit precedence rule supplied in your prompt.
Do not reward a confident answer when the evidence is ambiguous. A production workflow often needs the model to expose uncertainty.
5. Measure the cost of an accepted task
Record more than the advertised token price:
| Measurement | Why it matters |
|---|---|
| Input and output usage | Basic consumption |
| Number of requests | Multi-turn work adds calls |
| Retries | Failed attempts may consume resources |
| Completion time | User waiting time |
| Manual corrections | Operational effort |
| Accepted outputs | Useful delivery |
Use the provider’s actual billing records where available. Do not infer charges solely from visible response text.
A lower unit price may be attractive, but your application needs a cost estimate for completing its own work. Run the same task set and identify where extra rounds or corrections occur.
6. Move through an API gateway only after the tests are clear
An API gateway can provide an access route for multiple models. Your application still needs to validate model-specific behavior.
When evaluating TreeRouter, check which candidate models and protocols are currently available. Run the same acceptance fixtures through the chosen route and compare the returned fields, tool behavior, errors, and recorded usage.
Do not assume a gateway will preserve every provider-specific feature without testing it.
Start at TreeRouter, review access and pricing, and register once the required model route is confirmed. Use a limited evaluation workload before moving production traffic.
Frequently asked questions
Which Chinese LLM is best?
The useful answer depends on the task and exact model version. Publish the fixture and acceptance criteria before declaring a winner.
Is OpenAI compatibility enough for migration?
It is a starting point. Structured output, tools, streaming, and model-specific options still require validation.
Should I test every available model?
Begin with a few candidates that meet your functional requirements. Expand only when a result or limitation justifies more testing.
A reliable migration decision is supported by repeatable tasks, inspectable outputs, and measured usage.
