Agent Reliability Lab

Overview

Agent Reliability Lab explores a useful boundary for agentic software: a language model can interpret a request and propose a tool call, while deterministic application code decides whether that call is valid, bounded, and safe to execute. The project is an independent engineering demonstration built around synthetic commerce data, not a client engagement or production commerce system.

The system uses Alibaba’s ReactAgent for the reasoning loop and Spring AI OpenAI-compatible adapters for DeepSeek and Grok. The model can inspect products, inventory, and orders. Inventory reservation follows a separate protected path: the agent can produce a proposal, but the application requires a confirmation token and idempotency key before it commits a stock change.

Engineering Challenge

An agent may emit malformed arguments, propose an unknown tool, report success without a successful operation, or repeat an action. Treating generated text as business truth makes those cases difficult to contain. The engineering goal here is narrower and testable: keep tool contracts and state changes in application-owned code, and make the observable outcome traceable to validated tool results.

Architecture

The Spring server exposes chat, confirmation, provider-status, health, and trace endpoints. An orchestration service resolves an explicit provider and creates a request-local agent. Registered tool callbacks validate arguments and execute against deterministic domain state. SQLite stores synthetic catalog and reservation/idempotency state; traces are bounded in-memory snapshots. A separate evaluation module runs scripted scenarios. The React demo consumes the API and does not own authorization or mutation decisions.

Key Design Decisions

1. Application code owns tool contracts. Strict argument validation, business constraints, and tool results are handled outside the model. Unknown tools and malformed requests fail explicitly.
2. Protected writes require a separate confirmation request. A proposal alone does not change stock. The confirmation endpoint consumes an application-issued token and uses an idempotency key to prevent duplicate effects. The demo token is not bound to an authenticated identity.
3. No provider fallback. Provider selection and credential errors are surfaced rather than silently routing requests elsewhere.
4. Evaluation separates contracts from model quality. Offline tests use scripted models for repeatability and do not make paid provider calls. Their results describe application behavior, not autonomous model accuracy.

Implementation and Verification

The repository separates reusable agent/domain logic (arl-core), Spring HTTP composition (arl-server), and offline evaluation (arl-evaluation). Its documentation describes model-call and tool-proposal bounds, error sanitization, immutable provider configuration, and trace events for model calls and tool execution. The frontend provides provider status, chat, protected confirmation, and trace inspection.

The README documents 59 deterministic offline scenarios spanning 10 reliability categories. Tests cover tool behavior, API paths, persistence recovery, provider configuration, and evaluation. The most recent visible GitHub Actions quality run for the inspected main commit succeeded. This is repository workflow evidence, not a fresh local build performed for this site.

Limitations and Further Work

This is a local synthetic commerce lab. It has no authentication, durable audit trail, identity-bound authorization, or production deployment profile. The confirmation flow demonstrates separation of proposal and execution but is not a production human-approval system. Scripted tests cannot verify live model quality or provider availability. A production design would need to add identity, authorization, durable audit and operational controls appropriate to its domain.

Source

• GitHub repository
• README and setup instructions
• Architecture documentation