The question to ask about every side-effecting tool is brutally simple: what happens if it executes zero times, once, twice, or halfway? If the answer is unclear, the tool is not ready for production.
Models do not remove distributed-systems uncertainty. They make it easier to generate actions, which means retries, ambiguous timeouts, duplicate side effects, and partial failures matter even more.
Tool retry is not model retry
A transient 503 from a downstream service should normally be handled in the deterministic tool/runtime layer. There is no reason to ask the model to rediscover that it should retry a network request. By contrast, a semantic result such as “tracking number not found” may genuinely require model reasoning about what to do next.
Idempotency is the default for writes
Any side-effecting tool should be idempotent or sit behind an idempotent service. The client generates or reuses a stable operation ID. If a timeout occurs, the system can retry without creating a second refund, a second ticket, or a duplicate CRM update.
1operation_id = 8f1c...2 ↓3write request4 ↓5timeout6 ↓7retry with SAME operation_id8 ↓9backend returns original resultTimeout does not mean failure
If a downstream call times out, the true state may be success, failure, or unknown. A robust system represents UNKNOWN explicitly and tries to reconcile against authoritative state. Blindly converting timeout to FAILED is how duplicate side effects happen.
Exactly once is usually a contract, not a wish
If the downstream system supports an idempotency key or a queryable operation status, safe retry is straightforward. If it supports neither, you may have to choose between at-most-once delivery (duplicates avoided, work may be lost) and at-least-once delivery (work eventually happens, duplicates possible unless the consumer deduplicates). Do not claim exactly-once semantics without the primitives to implement them.
Partial failure should be reported precisely
Suppose Salesforce updates successfully but Slack notification fails. Do not roll back the CRM simply because notification failed, and do not tell the user the entire operation failed. Preserve the successful state, retry the notification according to policy, then report exactly what succeeded and what did not.
PFPLabs takeaways
- Make write tools idempotent.
- Represent ambiguous outcomes as UNKNOWN, not FAILED.
- Retry transient infrastructure failures below the model layer.
- Treat partial success as a first-class state. Truthful user experience depends on authoritative backend state.