Why GLM Vision API Misreads Small Screenshot Text
A GLM vision request can identify a settings page correctly and still misread the tiny account identifier in its corner. The screenshot arrived, the model understood the overall scene, and the answer sounds reasonable. Yet the one string your application needs is wrong.
Treat this as a visual-input design problem. Before changing prompts or models, establish which pixels reached the cloud API, how much context surrounds the target, and whether the original image contains readable characters. This clinic develops an overview-plus-crop workflow for small screenshot text. It does not report a benchmark or promise that cropping always improves recognition.
The running example is an illustrative support screen: a 3840-by-2160 screenshot contains an error panel, a short identifier, and a status label. The goal is to transcribe the identifier and explain which panel it belongs to, while leaving unreadable characters unresolved.
Start with the visual question, then select the API
“Describe this screenshot” and “read the identifier beside the failed job” are different tasks. The first can succeed using large shapes and familiar interface patterns. The second requires individual characters plus enough surrounding layout to associate them with the correct record.
Write the acceptance condition before choosing a model: preserve case and punctuation, identify the source region, and return an explicit unreadable result when the pixels do not support an answer. A plausible identifier is a failure if it was reconstructed from context.
The current GLM-5.3-Flash/FlashX documentation lists image input and documents multiple image_url blocks in a message. That makes it a documented upstream candidate for sending an overview and a detail crop together. The model identifier available through your account or gateway still requires confirmation; a general GLM text integration does not establish vision support.
Use the Chinese LLM gateway selection guide for the broader access decision. Here, the decisive capability is narrower: the selected cloud route must deliver both image blocks to a vision-capable model without silently reducing the request to text.
Separate image access from image readability
Inspect the exact asset your backend will send. A preview thumbnail in your application may differ from the original upload. A signed URL may resolve to an image for your browser but expire before a remote service fetches it. A storage link can return a login page or redirect instead of the intended raster.
For a URL-based pilot, confirm that the intended remote fetch can retrieve the image without relying on your browser session. Keep the URL valid for the request’s actual lifetime. Avoid making private support material permanently public just to simplify a test; use a non-sensitive fixture and an approved delivery method.
The GLM model guide also documents a Base64 Data URL as an image input. That can remove an external fetch dependency when the selected route accepts it, but it does not repair blurry text. Encoding the same low-resolution thumbnail differently preserves the same missing information.
The Chat Completion reference defines the upstream request envelope. Check that envelope and the model-specific image instructions together. A URL included inside an ordinary text paragraph is not equivalent to a typed image content block. The WorkBuddy troubleshooting guide is useful when an application accepts text requests but its image path fails.
“More detail” begins with pixels, not an assumed flag
Do not copy a detail: high option from another provider and assume GLM interprets it the same way. The GLM image instructions consulted for this workflow document image blocks and their URLs; they do not establish a universal detail switch across every GLM route. Confirm optional parameters against the exact model and interface you will use.
Instead, make a small visual inventory. Record original dimensions, uploaded dimensions, image format, orientation, and the approximate height of the target characters. Compare the original with any transformed asset created by your application. If the original contains clean text but the transmitted thumbnail does not, fix that transformation first.
Prefer retaining the original pixels around the target over repeatedly exporting a compressed preview. Enlarging a crop can make it easier for a person to inspect, but interpolation cannot recover letters that were already blurred together. A screenshot taken at the wrong scale may need to be recaptured at its source.
Avoid destructive “cleanup” before establishing a baseline. Aggressive sharpening, thresholding, or color removal can alter punctuation and thin strokes. Keep the unmodified source available, and treat each transformation as a separate input version with its own evaluation result.
Build a crop that preserves meaning
A useful crop includes the target text, its field label, and enough adjacent interface to identify the record. Cropping tightly to a string can remove a minus sign, truncate the first character, or detach a value from its label. Cropping an entire dashboard again defeats the purpose.
For the illustrative screen, keep the full screenshot as image A. Make image B a crop around the error panel, including its heading and the identifier row. Tell the model that B is a detail from A, not a second independent observation. Ask it to use B for transcription and A for page context.
Maintain a crop manifest containing the source dimensions, source asset identifier, and crop rectangle in original-image pixels. The rectangle should use a documented origin and boundary convention. If a later stage resizes the crop, record that transformation separately. Otherwise, a returned location cannot be reliably shown on the original screen.
The following Python example plans a crop and constructs an unsent message. It uses synthetic coordinates and placeholder asset URLs, performs no image manipulation, and makes no network request. The coordinate convention is left, top, right, bottom, with the right and bottom boundaries excluded. This matches the rectangle convention documented by Pillow’s image operations.
import math
def crop_plan(width, height, region, padding=32):
if type(width) is not int or type(height) is not int:
raise ValueError("integer source dimensions required")
if width <= 0 or height <= 0 or type(padding) is not int or padding < 0:
raise ValueError("invalid dimensions or padding")
if len(region) != 4 or not all(math.isfinite(v) for v in region):
raise ValueError("four finite normalized coordinates required")
left, top, right, bottom = region
if not (0 <= left < right <= 1 and 0 <= top < bottom <= 1):
raise ValueError("region must lie inside the source")
box = (
max(0, math.floor(left * width) - padding),
max(0, math.floor(top * height) - padding),
min(width, math.ceil(right * width) + padding),
min(height, math.ceil(bottom * height) + padding),
)
return {"source_size": [width, height], "crop_box": list(box)}
def vision_payload(model_id, overview_url, crop_url, plan):
if not model_id or not overview_url or not crop_url:
raise ValueError("confirmed model and two asset references required")
task = (
"Image A is the full screenshot. Image B is a crop of A. "
f"Source dimensions and crop rectangle: {plan}. "
"Read the identifier and status in B; use A only for page context. "
"Preserve case and punctuation. Do not complete uncertain letters. "
"Return the observed text, source image, and unreadable characters."
)
return {
"model": model_id,
"messages": [{"role": "user", "content": [
{"type": "text", "text": task},
{"type": "text", "text": "Image A: overview"},
{"type": "image_url", "image_url": {"url": overview_url}},
{"type": "text", "text": "Image B: detail crop"},
{"type": "image_url", "image_url": {"url": crop_url}},
]}],
}
plan = crop_plan(3840, 2160, (0.5, 0.25, 0.75, 0.5))
assert plan["crop_box"] == [1888, 508, 2912, 1112]
assert crop_plan(100, 100, (0, 0, 0.1, 0.1))["crop_box"] == [0, 0, 42, 42]
request = vision_payload(
"CONFIRMED_VISION_MODEL_ID",
"https://example.invalid/overview.png",
"https://example.invalid/detail.png",
plan,
)
assert sum(p["type"] == "image_url"
for p in request["messages"][0]["content"]) == 2
print("offline crop and message fixtures passed")The planner does not locate the identifier automatically. Its region comes from your interface metadata, a reviewer, or a separately validated localization step. The placeholder URLs intentionally cannot serve images. Before a live call, materialize the crop, inspect it, replace both references, and set the confirmed model ID.
For live deployment, follow the authentication and endpoint rules in the official HTTP guide, or the corresponding documented gateway route. No live request is executed here. The GLM integration guide supplies surrounding client context; its text examples should not be treated as proof of image support.
Read the answer against the image, not its confidence
A readable-looking answer is not sufficient evidence. Compare every identifier character with the original source, including similar forms such as zero and capital O. Preserve the model’s raw transcription before applying application normalization. A downstream rule that removes spaces or changes case can hide a recognition error.
For the pilot, run three input variants on the same labeled examples: overview only, crop only, and overview plus crop. Keep the requested field, model, and evaluation rules fixed. The experiment asks whether the added visual context improves your task, not whether one response sounds more detailed.
Track exact field matches, wrong-record associations, invented characters, and appropriate unreadable responses separately. A crop that improves transcription but selects the wrong row is not a successful design. Extend the Chinese LLM acceptance tests with these image-specific criteria.
When the overview and crop disagree, show the disputed region for review. Do not silently choose the longer answer or merge both strings. If neither image supports a character, request a better capture or return an unresolved field. Repeated calls on the same inadequate pixels do not create independent evidence.
Review the crop at the point of failure
When a field fails, inspect the crop as an independent artifact. Imagine that the original row says AB-10O, but the response says AB-100. First ask whether the last character is distinguishable in the delivered pixels. If it is not, neither the field format nor a familiar account pattern should be used to manufacture certainty.
Then inspect the neighborhood. A second identifier one row below may explain a wrong-record answer. A label outside the crop may explain why the model read a timestamp instead. Correct the crop boundary for that reason, document the change, and rerun the same examples. Do not simultaneously alter the model, prompt, image format, and crop size; that makes the improvement impossible to attribute.
For a fixed application layout, region definitions can be versioned alongside the interface. Include a screen-size or layout identifier so a desktop crop is not reused blindly on a narrow mobile page. When the layout is unknown, a human selection can be a better initial workflow than an unvalidated automatic locator.
These are proposed engineering controls, not reported model behavior. Their purpose is to make a mistaken character explainable: unavailable pixels, inadequate resolution, lost label context, incorrect region, or recognition failure despite a readable source.
Know when OCR is the better first stage
Use the vision model when spatial relationships matter: which warning belongs to which card, whether a control is visibly disabled, or how text relates to a chart. If the job is simply to transcribe many text regions with layout information, evaluate a dedicated OCR service before building a conversational extraction loop.
Z.AI’s GLM-OCR guide describes screenshot and document text recognition. Its Layout Parsing API exposes a separate operation with extracted content and layout fields. That is a different API contract, not a model-name replacement inside the preceding chat payload.
A practical hybrid can use OCR to propose literal text and a vision call to resolve the relevant panel or interpret a visual relationship. Keep both outputs attributable to their stages. An OCR transcript does not prove a status icon’s meaning, and a vision explanation should not silently overwrite uncertain characters.
Confirm access to each required operation before choosing that design. Treerouter support for the particular vision route, multiple images, Data URLs, or the separate OCR operation must be checked for your account. None follows automatically from access to a GLM text model.
Turn one readable screenshot into a useful cloud pilot
Choose a small, permission-approved set covering the screens your application actually receives: dense desktop text, mobile captures, mixed Chinese and English labels, and deliberately unreadable examples. Include the exact image preparation path your users will exercise. A handcrafted crop that bypasses your upload system cannot validate the deployed workflow.
If a support agent or coding harness will consume these images, also verify that it preserves the two image blocks and their labels. The agent harness integration guide helps separate client behavior from model capability. The crop manifest should remain application data even if the harness rewrites the prompt.
Bring the overview, crop, expected transcription, and unsent request shape to a Treerouter evaluation. Confirm the available model ID and exact vision contract, then make one controlled call before expanding the pilot. The useful milestone is an identifier your application can trace back to readable pixels, with an honest unresolved result when it cannot.
Learn more: Treerouter.






