Runtime

Runtime determines where a workload executes and which resources it can reach. A long-running agent may use tools and write files while holding a session credential, so it needs more than a request-response process.

Workload types

Request-response services accept a user request and return a model-generated answer. Existing web platforms can support them with streaming and timeouts sized for model latency.

Batch and scheduled jobs cover corpus classification, nightly summarisation, and enrichment pipelines. The model call is one step in the job.

Long-running agents are stateful sessions that may last for minutes or hours. They hold credentials, execute code, write to a filesystem, and may need interruption or resume. Platforms built around stateless twelve-factor services may need a separate runtime pattern for them.

Serverless timeouts can push an agent onto a manually managed VM. Treat that VM as production infrastructure with an owner and the same controls as the rest of the runtime.

Agent runtime requirements

• Provide a filesystem for the session and destroy it at session end unless an approved retention requirement applies.
• Isolate generated code from the host and other tenants. Treat instructions in web pages and retrieved documents as untrusted input.
• Restrict outbound network access to the destinations required by the task.
• Inject credentials at runtime, scoped to the session and expiring with it.
• Let a human stop the session and release its work and credentials.
• Emit a trace of model and tool activity during the run.

Runtime blast radius

Record the network, cloud role, reachable databases, and environment credentials for each workload. These define the effect of compromised or misdirected execution.

Do not let host placement grant an agent production write access by accident.

Developer-facing runtime interface

Railway, Render, Heroku, and cloud application services give developers a short path from source code to a running URL. An internal platform needs a similarly direct workflow if teams are expected to use it.

Present a small interface over the existing infrastructure: deploy, obtain a URL, view logs, attach a database, and request a secret. Kubernetes, ECS, and VMs can remain implementation choices behind that interface. This separation supports portability.

Tenant isolation

Even inside one company, AI workloads are multi-tenant: different teams, different data classifications, different model access. A shared runtime needs an explicit isolation policy before it hosts workloads with different trust levels.

Choose the isolation boundary before workloads are deployed: namespace, node, account, or cluster. A later change requires a workload migration.