Learn more：https://treerouter.com

Abstract

Retrieval-Augmented Generation (RAG) has become a standard approach for domain-specific intelligent assistants, yet the engineering overhead of implementing function calling, message protocol matching, exception handling and retrieval threshold control often slows down delivery cycles. This article documents a real-world production case: leveraging the TRAE Agent mode to develop the RAG tool routing module for a Warehouse Management System (WMS) intelligent assistant. The module supports natural-language inventory lookup and operation manual retrieval. The record covers requirement definition, one-shot code generation, validation testing, failure analysis, prompt tuning and final takeaways, with measured runtime metrics and retrieval similarity scores preserved for reference. The case illustrates the boundary between what AI coding agents can automate reliably and the domain-specific calibration work that still requires human engineers.

1. Project Background: Fixed Release Deadline for a Production WMS Assistant

The WMS intelligent assistant in this project was scheduled for public rollout on November 7. Its core user-facing capability allows new warehouse staff to query stock levels and browse standard operation manuals using natural language, without navigating complex menu systems or reading lengthy static documents manually.

Before introducing TRAE Agent, the engineering team had spent ten consecutive development days building foundational components. Completed modules included streaming output, function calling stacks, conversation context management, automatic retry for API exceptions, embedding pipelines, parent-child chunk retrieval logic, and baseline anti-hallucination controls for RAG workflows. At this stage, the core RAG question-answer script, saved as day-10.py, had passed end-to-end validation and returned stable retrieval results.

When TRAE released its dual-end integrated update combining Agent and IDE modes, the team decided to offload the 11th-day development task to the agent. The target module was RAG tool registration and autonomous routing logic, which would let the large language model decide when to trigger retrieval tools according to user prompts. The team also set out to evaluate how capable this AI coding tool is when faced with strict production constraints rather than trivial demo scripts.

The requirements were formalized into three non-negotiable rules built on top of the verified day-10.py base:

  1. Register two independent callable tools: query_stock for real-time inventory inquiry, and search_manual for operation manual lookup. The search_manual function reused the mature parent-child chunk retrieval logic implemented on day 10.
  2. Implement a two-jump main loop. On the first jump, the model evaluates user input and determines whether to invoke registered tools. Once tool execution completes and structured results return, the second jump generates the final natural-language response based on retrieved payloads.
  3. Enforce strict message protocol matching. assistant(tool_calls) messages and corresponding tool response messages must form strictly paired records. Any mismatch triggers an HTTP 400 error from the backend API layer.

This third rule was a hard-won lesson from earlier manual debugging. Mismatched tool call identifiers are a common failure source in function-calling workflows, and inconsistent pairing will directly break the entire conversation session.

2. Single-Prompt Task Briefing and One-Shot Code Generation

Instead of splitting the requirement into smaller subtasks and feeding them sequentially, the engineer submitted the full requirement statement directly to the TRAE Agent. The prompt explicitly highlighted the protocol constraint about paired tool messages, as a stress test to verify whether the agent understood the operational risk behind this API contract rule.

The agent produced the complete target script day-11.py in a single generation pass, with approximately 200 lines of functional code. Manual review revealed that the output was not merely a template-based code skeleton. It incorporated defensive programming patterns that the engineer had learned and implemented incrementally over the prior ten days of coding, pre-emptively avoiding pitfalls that had caused repeated bugs during manual development. Three design choices stood out during review:

First, the agent respected the API protocol strictly. It avoided the naive approach of inserting raw API return payloads back into the message array. Instead, it manually reconstructed the tool_calls object before appending it to the message list, with inline comments marking this structure as the anchor point for subsequent matching tool messages. The implementation guaranteed that each unique tool_call_id maps to exactly one corresponding tool response entry. This was the exact fix the engineer had applied repeatedly after hitting 400 errors in earlier iterations.

Second, three layers of exception fallback were embedded into tool execution logic. The handler captured JSON parsing failures for incoming parameters and substituted safe default values. It intercepted requests for undefined function names and returned a standardized unknown tool response. It also wrapped function runtime exceptions, converting crash traces into structured result payloads fed back to the LLM. This last point aligns with a core production rule established in week one of the project: tool errors must always be returned to the model as valid results. Silent failure suppression often leads the LLM to invent content to fill missing data, which amplifies hallucination risks. The agent adopted this defensive pattern without additional prompting.

Third, the agent disabled the tool list for the second response jump by setting tools=None. If the tool schema remained attached in the second round of messages, the model could trigger chained, redundant tool calls even after sufficient information had been retrieved. This subtle optimization is frequently overlooked in handwritten agent loops, and the agent included it automatically.

A Treerouter instance can simplify observability for these multi-turn tool calls, consolidating request logs and latency metrics across model and tool backends in a single view.

3. Functional Validation and an Unanticipated Edge-Case Failure

Four test prompts were prepared for validation after the script launched. The first three test cases passed with correct routing and reliable outputs:

  1. Query: *How much mineral water remains at bin WH001?* → The agent correctly dispatched query_stock, returning a stock count of 1200 units.
  2. Query: *What should be done when incoming goods arrive damaged?* → The agent invoked search_manual. The retrieval pipeline matched the *Inbound Receipt Process* document with a similarity score of 0.58, and the generated answer matched the official manual content.
  3. Query: *How to handle inventory counting discrepancies?* → The agent invoked search_manual, matching the *Inventory Count Specification* with a similarity score of 0.602, and produced an accurate, grounded response.

The fourth test case exposed a critical weakness. The user prompt was: *What is the return goods workflow?* The operation manual repository contained no dedicated document describing return workflows. The nearest matching entry returned by vector search was the *Outbound Process*, with a similarity score of 0.609. The preconfigured rejection threshold was set to 0.45. Since 0.609 exceeded this cutoff, the code allowed the retrieved chunk to flow into the LLM context window.

The model then combined the unrelated outbound process text with invented reasoning. The response stated that no dedicated return workflow record was found in the manual, then appended fabricated procedural steps including return application intake, item inspection, goods processing and refund settlement. The first clause was truthful, but the subsequent procedural content was entirely hallucinated.

This failure was not a flaw in the code generated by TRAE Agent. The similarity threshold value of 0.45 was defined by the human engineer and copied faithfully into the script. The semantic overlap between outbound and return workflows existed within the vector embedding space, a subtle domain-specific phenomenon that can only be identified by running test queries against real production vector data. The threshold filter effectively blocks results with very low similarity scores below 0.45, but it cannot distinguish ambiguous grey-area matches between 0.45 and 0.65 where semantic similarity is superficial rather than factually relevant.

4. Prompt Tuning: Fixing Hallucination Without Modifying Core Code

The resolution required zero changes to the Python routing and retrieval codebase. The fix was delivered by adding one explicit rule inside the system prompt template.

  • Original implicit rule: rely solely on similarity score filtering to decide whether retrieved content is usable.
  • Updated explicit rule: if retrieved matches do not align semantically with the user question (such as returning outbound process content for a return workflow inquiry), the response must state that no matching content exists in the manual. The model is prohibited from inventing workflows, steps or speculative procedures to fill gaps.

After re-running the fourth test prompt with the revised system prompt, the model returned a concise, safe statement: *No relevant content about return workflows exists in the manual.*

This single prompt adjustment also created a beneficial side effect. Responses for the first three valid test cases automatically appended source citation tags formatted as [Source: xxx]. Constraints written into system prompts often reinforce one another; this emergent citation behavior was an unplanned positive outcome of tightening grounding rules.

The full end-to-end cycle from requirement submission to passing all four validation cases took less than one hour of wall-clock time. Measured TRAE Agent execution time was 4 minutes and 13 seconds across 15 interactive command iterations, including context retention and self-check steps. The saved engineering time was redirected to building evaluation datasets, the component that remains impossible to fully delegate to AI agents.

5. Findings and Practical Engineering Takeaways

This hands-on trial clarifies the appropriate role of AI coding agents in production software delivery. These tools excel at encoding mature engineering guardrails, but they cannot automatically understand the nuanced semantics of private domain data.

The defensive programming implemented in the TRAE-generated script outperformed the engineer’s initial handwritten version. Exception wrapping, parameter validation, strict tool-call protocol matching and loop state management all reflect collective best practices absorbed from large volumes of public code repositories. The agent reliably encodes these established patterns when requirements are clearly stated.

However, the agent cannot resolve edge cases rooted in domain semantics. It cannot know ahead of time that outbound workflow embeddings are close to return workflow embeddings within this warehouse dataset. These boundaries require human calibration with domain test samples.

The conclusion drawn from this deployment is practical and actionable. AI agents can generate production-ready code for standard agentic routing workflows in a single pass. Engineers retain ownership over domain-specific definitions: similarity thresholds, rejection criteria, evaluation benchmark sets and semantic boundary rules. The agent supplies industry-standard defensive implementation, while the human engineer builds the reference dataset that defines what “relevant” means for the business.

For teams scaling multiple LLM service endpoints, consistent threshold and prompt rule management becomes increasingly important. Centralized routing helps keep grounding rules aligned across assistant deployments.

The technical stack for this case was TRAE CN latest release with Agent mode, running on Windows, Python 3.x and GLM-4-Flash. All code produced by the agent was reviewed and adjusted manually before entering production traffic.

6. Conclusion

Agentic coding platforms such as TRAE reduce the repetitive engineering burden of building RAG tool routing systems. In this WMS assistant project, the agent delivered a complete two-jump function-calling loop with multi-layered exception handling and protocol compliance built in. Automated generation cuts down development cycles significantly for standard agent patterns, but it does not remove the requirement for human-led domain validation.

Similarity thresholds and prompt grounding rules are the control knobs that separate safe, factual RAG outputs from hallucinated responses. Even well-written routing code will fail silently on semantically ambiguous vector matches unless benchmarked against real user questions and private document corpora. The most efficient workflow combines AI generation of boilerplate defensive infrastructure with human engineers designing targeted test suites and semantic rejection rules. This division of labor balances delivery speed with the reliability required for business-critical warehouse systems.