Kimi K3 vs Qwen3.5-Flash: The 100x Price Test

Abstract

This benchmark reveals a counterintuitive result: Kimi K3, one of the highest-priced large language models on the market, underperformed the low-cost Qwen3.5-Flash in a structured documentary subtitle translation task. Kimi K3 is priced at ¥20 input / ¥100 output per million tokens, while Qwen3.5-Flash costs only ¥0.2 input / ¥2 output per million tokens, creating a 100-fold price gap. Despite this massive cost difference, Qwen3.5-Flash achieved an overall score of 0.76, versus Kimi K3’s 0.52. This article presents the complete reproducible benchmark, including a seven-dimensional scoring framework, judge rationale, raw model outputs, and full pipeline setup. All testing was conducted via Treerouter compatible endpoints.

1. Benchmark Task Definition: Translate 12 Subtitle Segments for Documentary Translators

The task was carefully defined to simulate the real workflow of professional documentary subtitle translators. The assigned role adheres to strict rules: preprocess character counts before polishing text, treat line length limits as hard constraints rather than loose recommendations, and keep narration and character dialogue linguistically separated. The objective was translating 12 English subtitle entries from a fictional documentary *The Freezing Line* into Simplified Chinese. Each entry must retain timing markers, and the test set included a noisy machine-translated draft as interference material, which the model must not copy directly.

Three layers of difficulty were built into this evaluation:

1. Line length constraint: Each subtitle line has a strict character limit. Speaker tags consume part of the character budget, forcing models to count visible characters instead of simply summarizing text.
2. Register differentiation: Narrator monologue and fisherman casual dialogue must be distinguished. The raw line *“No ice, no rest. No rest, no crab.”* is a short oral chain; direct literal conversion into formal written prose would break the original tone.
3. Anti-interference test: A flawed machine translation draft was embedded into the prompt, to check whether the model blindly copies incorrect preliminary translations.

The judge model used in this test was gpt-5.6-sol, not part of the competing models. Evaluation covered seven independent dimensions, each scored on a scale from 0 to 1: accuracy, completeness, fluency, format compliance, judgment, register, and terminology. The multi-dimensional design prevents single-dimension bias: even if translation text reads smoothly, a broken format will deduct points heavily. The judge prompt required explicit citations from source text for every scoring decision to reduce subjective impression-based grading. While the judge model carries inherent preference bias, all scores in this report follow its assessment.

2. Pipeline Implementation: How the Two Translation Workflows Were Built

Both model runs were executed through compatible API endpoints. The core request code is shown below:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TREEROUTER_API_KEY"],
    base_url="https://treerouter.com/v1",
)

resp = client.chat.completions.create(
    model="qwen3.5-flash", # replace with kimi-k3 for the second run
    messages=[
        {"role": "system", "content": ROLE_PROMPT}, # Subtitle translator role definition
        {"role": "user", "content": TASK_WITH_SRT}, # 12 subtitle lines + noisy draft
    ],
)

Model identifiers used the stable aliases from the marketplace detail page (qwen3.5-flash and kimi-k3), rather than raw version strings. The pipeline supported resumable batch execution: outputs were saved to files named by target model. Completed tasks were skipped automatically on reruns, avoiding redundant generation if the evaluation phase crashed. Response headers retained X-Request-Id following the REQ-xxxx convention for troubleshooting mismatched scoring records.

The parsing logic directly segmented content by standard SRT structure. Blank lines separate individual entries; each entry contains a sequence number on line one, timestamp axis on line two, and subtitle body in remaining lines. This parsing method became the key point of failure for Kimi K3.

3. Benchmark Results: 0.76 vs 0.52

The total score gap of 0.24 mainly comes from three dimensions: completeness (0.86 vs 0.32), format compliance (0.60 vs 0.03), and accuracy (0.78 vs 0.58). Notably, terminology scores stayed nearly balanced at 0.82 for Qwen3.5-Flash against 0.76 for Kimi K3. Kimi K3 was fully capable of terminology translation, but failed severely at structured delivery. On latency metrics, Qwen3.5-Flash finished in 86.98 seconds, while Kimi K3 required 169.1 seconds, nearly double the runtime despite its much higher price.

4. Failure Analysis: Why Kimi K3 Scored Only 0.03 on Format Compliance

The judge’s original comment summarized the root failure clearly: the model failed to produce 12 matching sequence numbers and timestamps. Its output began with reasoning text such as *“Let me work through this carefully”*, completely violating the required delivery structure.

A snippet of raw output demonstrates the flaw:
> Let me work through this carefully. I need to translate 12 subtitle lines from English to Simplified Chinese for a documentary called The Freezing Line… (content truncated)

Essentially, Kimi K3 exported its internal reasoning process as the final deliverable. This is not a translation capability limitation. Its terminology score of 0.76 shows consistent and accurate handling of domain vocabulary, including proper nouns such as “Skerry Gulf”, “Thomas Berg”, “quota”, and “fishing catch”. It also avoided mistranslating the word “bank” as a financial institution.

The low completeness score of 0.32 stems directly from this structural defect. When parsed under SRT rules, the introductory analytical text is treated as invalid entry content. The 12 required IDs and timestamps were not formatted properly, causing automatic SRT parsers to fail immediately. If fed into subtitle rendering pipelines, the whole batch would trigger parsing errors.

The judge also noted a terminology weakness for Kimi K3: entry 3 did not fully capture domain context for nearshore shoals and fishing moratorium. The word “bank” here refers to a fishing shoal, not a financial bank. Qwen3.5-Flash also partially missed this nuance, which will be covered in later analysis.

5. High-Performing Candidate: Where Qwen3.5-Flash Wins and Its Remaining Weaknesses

Qwen3.5-Flash’s biggest strength was completeness at 0.86. The judge noted all twelve entries contained complete translations; numeric values including thirty-one years, one-third, two days per week and monthly cycles were preserved without loss. Its handling of the oral dialogue chain showed strong register awareness, contributing to its register score of 0.79.
> 4 00:00:14,100 --> 00:00:16,700 [Thomas] No ice, no rest. No rest, no crab.

The parallel rhythm of the original sentence *“No ice, no rest. No rest, no crab.”* was preserved in translation. Even so, Qwen3.5-Flash was not a perfect solution. Its weakest metric was format compliance at 0.60. The judge’s feedback: sequence IDs, timestamps and entry counts were all correct, and most text kept within two lines. But speaker tags pushed some lines past the 16-character cap.

The model treated the hard line-length constraint as a recommendation, rather than an unbreakable rule. This is a mechanical formatting defect, which can be fixed with lightweight post-processing scripts. Two accuracy deductions were recorded: entry 1 lost the “harbour” information, and entry 10 failed to fully convey the original meaning of ship boarding.

This reveals a critical practical nuance. The prompt explicitly specified line length as a hard constraint, yet Qwen3.5-Flash still relaxed the requirement. For production use, outputs still need an additional character-count validation pass.

6. Model Selection Guidance: Inferences Drawn From Score Breakdown

We can derive practical deployment recommendations directly from this benchmark dataset:

• Do not use Kimi K3 to generate final deliverables directly. The format score of 0.03 means outputs almost always require manual restructuring, unless the workflow enforces structured output modes such as tool calls or JSON mode before converting results to SRT subtitles.
• Qwen3.5-Flash works well for subtitle first drafts. Its completeness score of 0.86 proves information retention is reliable. Format violations such as line overflow can be resolved automatically with a simple script that wraps text at the 16-character threshold.
• Terminology performance between the two models is comparable (0.82 vs 0.76). The gap in specialized vocabulary cannot justify the 100x price difference. For subtitle translation workloads, the cheaper option delivers superior value.
• Latency gap (86.98s vs 169.1s) becomes amplified for batch processing. A full documentary with hundreds of subtitle lines will nearly double total pipeline runtime with Kimi K3.

Important caveat: this is a limited two-run single-task benchmark. The sample size cannot prove universal superiority of either model across all workloads. It only reflects their performance under this exact task configuration. Both models resisted copying the flawed machine-translated draft, a positive outcome exceeding initial expectations.

One striking observation emerges from this subtitle translation benchmark: the biggest source of score loss was not translation quality itself. Failures centered on delivery format and numeric integrity. For subtitle translation and similar structured generation tasks, model performance hinges more on adherence to engineering constraints than linguistic ability. Future evaluations will embed line-length checking and automatic scoring into the workflow to catch constraint violations automatically.

When building multi-model translation pipelines, developers often need unified routing and credential management across different LLM endpoints. Treerouter serves as an API gateway to manage mixed model backends and standardize request logging.

All test runs were validated via Treerouter’s OpenAI-compatible endpoints, using model aliases qwen3.5-flash and kimi-k3. The judge model was gpt-5.6-sol. Full API parameter documentation can be found at treerouter.com/docs. Batch requests carry X-Request-Id headers for audit.

7. Conclusion

This benchmark demonstrates that raw model price is a poor predictor of performance on structured production tasks like documentary subtitle translation. The 100-times-more-expensive Kimi K3 showed solid terminology translation capability, yet catastrophic failures in output formatting ruined its overall score. Qwen3.5-Flash maintained high completeness and acceptable fluency, with only minor, script-fixable formatting defects.

For subtitle translation workloads, the key risk is not vocabulary accuracy. It is compliance with rigid structural rules, character count caps, and output schema. Expensive high-end models do not guarantee they will respect hard format boundaries. Teams can adopt low-cost flash models for first-pass translation, then add lightweight post-processing scripts to enforce SRT formatting.

For teams running multi-LLM workloads, an API gateway helps standardize access, track latency and token consumption across all model endpoints. Treerouter unifies access control and observability for mixed model services.

Learn more：https://treerouter.com