A diagnostics agent for a bank, Part 2: what should we build?

1. Start with the harness we already have

The Agent Workbench separates the workflow from the model transport. Its harness owns the run lifecycle, cancellation, tool-call budgets, evidence records, and saved state. A validated tool registry handles capabilities. The default model connection is hosted via VPC. That is a useful shape for a diagnostics product: changing the model should not change what it is allowed to read.

The implementation already groups documents, logs, and databases by application. Its database tools accept a restricted pre llm-inferred structured query, validate tables and columns, and compile parameterized SELECT statements. db_dictionary describes the data; db_query_plan previews the compiled query; db_query executes a bounded read. Local and remote log tools expose bounded searches and windows, with coverage and truncation information.

There is a significant boundary to preserve. Application selection is source scoping, not a bank employee's authorization. The workbench assumes a trusted operator, and its optional integrations have their own trust boundaries. A banking service needs authenticated individual identities, independently enforced entitlements like cyberarks etc , isolated tool execution, controlled egress, and protected audit storage. A saved DB conversation alone is not a tamper-resistant incident record.

2. What should the investigator be able to see?

I would connect logs, databases, endpoints, and infrastructure—but give each a purpose-built tool. Logs tell us what components recorded. Metrics show the shape of the failure. Traces help follow a request. Database observations reveal waits, replication lag, or business state. None is a complete account on its own.

The common foundation is identity and time: application, service, environment, region, release, request or trace ID, and a permitted business correlation key. OpenTelemetry's context propagation provides a standard way to connect traces and other telemetry across services. Legacy batch jobs and external payment rails may need explicit mappings instead. Preserve event time, collection time, timezone, and clock uncertainty; ordering by ingestion time can produce the wrong incident story.

I would ask the backend to aggregate, group, and narrow the time window before returning examples to the model. A dictionary can explain a field or log format, as Agent Workbench already does, but inferred meanings and sampled coverage must remain labelled. Missing results should say whether the source was quiet, unavailable, filtered, sampled, or truncated.

3. Private cloud and open weights are separate choices

There are at least three questions here. Who operates inference? What data can reach it? Can we obtain and run the model's weights ourselves? A proprietary model can be delivered through a private cloud endpoint; an open-weight model can still be consumed through someone else's public API. Neither label settles privacy, residency, retention, or quality.

For example, Amazon Bedrock supports private connectivity through VPC interface endpoints. That changes the network path; it does not turn the service into bank-operated compute. I would separately review processing locations, cross-region routing, abuse-monitoring terms, retention, request logging, encryption, and provider access for the exact service and model configuration.

Yes, there are downloadable open-weight candidates. Qwen3-32B is one concrete example whose published repository carries Apache 2.0. It is an example to evaluate, not a claim that it is the best model for a bank. Check the exact release and licence, including any fine-tune or quantized derivative. Available weights also do not establish the broader openness described by the Open Source Initiative's AI definition.

My initial preference would be a model gateway with an explicitly approved primary model and a separately tested self-hosted fallback. Route by data classification and task, and fail closed when no approved destination is available. Do not silently send sensitive evidence to a public model because the private service timed out. The fallback can also share the primary's cloud, network, or identity failure, so its independence needs testing.

Evaluate structured tool calls, numerical consistency, unfamiliar schemas, long noisy records, refusal behaviour, and willingness to say evidence is missing. Include inference cost, peak-incident concurrency, and the effects of quantization. I would begin with retrieval of current evidence and runbooks; training on raw incidents creates a separate data-governance problem and does not keep a model current by itself.

4. One harness per system, or one for the whole bank?

I would separate the software we maintain, the processes we deploy, and the authority of each investigation. One shared harness codebase can run many scoped investigations. Sharing a model-serving cluster does not require sharing their credentials or evidence. Conversely, a separate process for every application does not automatically isolate it.

A harness per application understands local vocabulary and ownership, but may miss the shared database, identity provider, or payment gateway causing several symptoms. One all-encompassing investigator sees more relationships, but invites broad permissions, context overload, and a large failure domain. The right boundary may be a business service spanning several applications, rather than one repository or one machine.

I lean toward common control and federated investigation: shared tool contracts, model policy, evaluations, and audit conventions; application or domain investigators with narrow access; and an incident coordinator that requests evidence across those boundaries. The coordinator receives permitted findings and provenance, not every investigator's secrets. A request to another domain is authorized again by that domain.

Confederation becomes useful language when legal entities or regions retain independent governance: they agree on an evidence contract while keeping local control. It is more expensive to operate. For a pilot, I would use one deployment of the existing scoped harness, one business service, and explicit domain connectors. Add distributed agents when ownership or isolation demands them, rather than starting with a swarm.

5. Should it be able to search the web?

Sometimes. A vendor advisory, a newly documented error, or a public status page can explain something our internal runbooks do not. I would still launch the first pilot without web search, then add it as a separate research lane. Private inference and web access are independent decisions; a locally hosted model can leak data through a search query.

Keep the internal tool runners without internet egress. A research broker accepts only reviewed, sanitized queries such as a public product name, version, and generic error. Customer names, account identifiers, internal hostnames, URLs with tokens, and pasted log fragments do not belong in a public search. An approval should show the exact outbound query, rather than ask vaguely whether internet access is allowed.

Prefer official vendor documentation, status pages, and advisories. Capture the URL, retrieval time, publication date when available, and the excerpt used. Treat web pages, logs, runbooks, and peer-agent messages as untrusted evidence: instruction-like text in them cannot authorize another tool or widen scope. OWASP documents indirect prompt injection; keeping content and authority separate needs execution controls, even with retrieval, fine-tuning, or a second model checking the answer.

A vendor's green status page is weak evidence about our particular route. Ask whether its region, product, and update time match the incident. External research should suggest a test; internal observations must establish whether that explanation fits.

6. Follow one payment incident across the evidence

Imagine payment confirmations timing out at 02:13. The first guess is that the database is overloaded. The investigator should be able to test that guess against a few bounded observations, rather than ask for all logs from the last day.

E1 is a deployment record: at 02:10, a release reduced the application's connection-pool limit. E2 is telemetry: pool acquisition waits rise on new instances, followed by gateway timeouts and older queue messages. E3 is a permitted database diagnostic view: available sessions and database waits do not show the corresponding saturation. E4 compares old and new application instances: the old instances remain healthier under comparable traffic.

That makes a pool-setting regression a useful leading hypothesis. It is not proof: traffic assignment may differ, metrics may be sampled, and a healthy partner probe may miss an intermittent failure on the actual payment path. The report should name those limitations and propose the next discriminating check. A different model repeating the hypothesis is not another independent observation.

Before any recommendation to replay work, ask a different question: which payments have already committed, which acknowledgements are missing, and which business keys prevent duplicates? Aggregate reconciliation views may answer this with less exposure than raw customer rows. The investigator can identify uncertainty; replaying payments remains a separate operational decision.

7. Read-only still needs a budget and a contract

Agent Workbench's structured query compiler is a useful starting point: the model chooses from reviewed operations and allowed schema, while code produces the SQL and binds. I would extend that with bank-approved diagnostic views, a data dictionary containing units and table grain, and database-side workload limits. A row limit controls returned data; it does not necessarily limit the work needed to find those rows.

Prefer replicas or diagnostic stores for business reads, while showing replication lag. Primary-only facts such as current lock waits need a separately authorized, narrowly constrained operational path. Apply timeouts, concurrency limits, cancellation, and query-cost controls appropriate to that engine. PostgreSQL's statement and lock timeouts are examples of independent controls; calling an account read-only does not make its queries cheap or its results non-sensitive.

Endpoints also need more than a generic fetch_url tool. Register the service, allowed destination, method, path, credentials, timeout, response size, and expected side effects. GET does not guarantee that a poorly designed endpoint is harmless. Use approved health checks or synthetic identities, never an exploratory payment submission or an arbitrary URL discovered in a log.

The endpoint broker should enforce destination policy after DNS resolution and for every redirect, resist DNS rebinding, and exclude metadata services and unapproved addresses. An internal probe may legitimately reach private addresses; the external research lane should not inherit that access. Keep service credentials outside the model and attach them only after authorization.

8. How should it alert without creating another alert storm?

I would keep established monitoring responsible for detecting urgent service impact. An LLM outage or an unfinished investigation must not delay a page. Error-budget burn rates can help distinguish immediate intervention from slower investigation; Google's SRE workbook explains using multiple windows and burn rates. Banking also needs explicit signals for money movement, reconciliation, security, and business deadlines, including low-volume services where percentage alerts are misleading.

The agent would attach an evidence brief to the existing incident and update it when the leading explanation, scope, or recommended next check materially changes. Group by the affected business service and suspected shared dependency, preserve links to each symptom, and expire stale findings. Send immediate impact to the owning on-call route, uncertain investigation results to the incident workspace, and non-urgent recurring patterns to a ticket. An unacknowledged incident still follows the established escalation policy.

A useful alert says: what customers or processes are affected; when it started; what changed; the leading hypothesis and alternatives; evidence links and freshness; missing coverage; the owner; and the next decision needed. Distinguish observed impact from inferred cause. A model's self-reported 95% confidence is not a calibrated probability and should not determine severity.

Could the agent discover an incident before the monitors do? Possibly, especially a business-state anomaly. I would first run those detections in shadow mode, measure missed incidents and false pages, and promote a reviewed detector into the monitoring path. Repeated summaries should not create repeated pages.

9. Put a human at the decision, with enough context

Human involvement should be designed into the investigation. The on-call engineer sets the question and checks evidence. The incident commander coordinates competing priorities and communication. A database, network, or security owner controls exceptional access in their domain. Existing change policy determines who can authorize remediation and when dual control is required.

Preapproved bounded reads can run without asking permission for every query. Expanding to another application or accessing sensitive rows needs an explicit scope decision. An expensive diagnostic or outbound web query can require a separate approval. Show the exact operation, destination, expected load, data exposed, expiry, and remaining budget. Approval belongs to the authenticated person's entitlements; it cannot lend authority they do not have.

If the agent prepares a rollback, the reviewer needs the target release, evidence, likely impact, prerequisites, verification steps, and reversal plan. Bind approval to that exact proposal and current state, and invalidate it when the target or evidence materially changes. Any executor should recheck authorization and preconditions. A chat message saying looks good is too ambiguous to be the whole control.

The UI matters here. I would show a timeline, evidence with source timestamps, ranked hypotheses with counterevidence, and a small next-decisions panel. Let the engineer challenge an assumption, narrow scope, stop the run, or take over. Preserve the original evidence and the human's correction; do not turn an unreviewed correction into global model memory.

10. What happens when the investigator fails too?

Extend Agent Workbench's saved sessions into a durable incident workspace. Keep the incident ID, participants, scope, model and policy versions, tool requests, source timestamps, evidence hashes or protected snapshots, truncation, hypotheses, approvals, and final decisions. Hashes help detect changes; they do not by themselves prove that an event was truthful or make storage tamper-resistant. Apply access controls and retention to the stored evidence, model-service logs, and exported reports as well.

Put the investigator and a useful evidence path outside the failure domain it is investigating. Cache approved runbooks and topology, show when sources or the model are unavailable, and let a human open the evidence without inference. Set per-incident and bank-wide limits on model requests, database work, and concurrent investigations; an incident storm must not turn the investigator into another overload source.

Recovery needs explicit semantics. Re-reading current metrics after a restart can change the answer, so record the new observation. Re-sending a page or executing an approved operation needs deduplication and state checks. Test failover, evidence-store loss, expired credentials, and stale approvals. A fallback that returns a confident summary without its missing-source warning is a failed recovery.

11. The first build I would actually attempt

I would start with one payment service and a small group of its engineers. Reuse the Agent Workbench model adapter, run events, source scopes, dictionaries, and structured-query approach. Add an incident record, identity-enforced tool broker, deployment history, metrics and trace connectors, and an evidence brief in the existing incident tool. Leave public web research and production changes outside this first release.

Replay historical incidents using only the data available at each stage; do not leak the final postmortem or a later runbook into the answer. Compare against the existing search tools and runbooks, a single-model baseline, and the proposed scoped design. Measure time to a useful evidence-backed hypothesis, unsupported claims, source coverage, false escalations, query load, cost, and the engineer's review time. Speed without correct evidence is not an improvement.

Include injected instructions inside logs, misleading timestamps, missing telemetry, schema drift, replica lag, malformed tool requests, permission failures, and model unavailability. A second model can critique an answer, but it does not replace these tests. Then run live shadow investigations, followed by opt-in assistance during real incidents. Agree acceptance thresholds and stop conditions with the service owners before judging the pilot.

12. What else could this become?

An incident memory could find a repeated failure that different teams named differently. A dependency view could reveal that three applications depend on one expiring certificate. A change-review assistant could compare a proposed deployment with earlier incident signatures. A reconciliation investigator could connect technical recovery with unfinished business work. Each is a distinct capability with its own evidence and access needs.

I am particularly interested in finding observability gaps: identifying where we cannot follow a payment, which owner is missing from the service map, or which runbook makes an unsupported assumption. That may be more valuable early on than a more elaborate agent hierarchy. Capacity forecasting, batch-deadline risk, vendor dependency analysis, and post-incident evidence packs are other possibilities, provided predictions and drafts remain clearly labelled and independently checked.

There are questions I would keep open. Does federation improve diagnosis enough to justify the latency and coordination? How often does external research change the next useful check? Can the bank-operated model handle the difficult incidents at acceptable concurrency? Which observations are safe to share across regions? Build the evaluation so these questions can change the design, rather than defending an architecture chosen before the first replay.