Deploy on Kubernetes with MAMS Infrastructure

MAMS Infrastructure runs ASTRA multi-agent systems built with MAMS across a cluster of machines, using Kubernetes. You describe a system once (which agents, how many, with what settings) and the platform deploys it, routes messages between its agents wherever they run, scales pools of agents with their load, restarts agents that remember their previous life, and shows what is running.

The platform is itself built with ASTRA and MAMS. The agents that deploy systems, serve the dashboard and watch for faults are ASTRA agents, and their interfaces are MAMS resources. Every pod runs an ams agent (a FIPA-style agent management service) that lists the pod’s agents as linked data.

This page describes MAMS Infrastructure 1.3.1, the current release (built on ASTRA 2.8.2 and MAMS 2.5.9). The project has its own installation guide, user guide, operations guide and upgrading guide, and the chart README lists every option.

How it fits together

Part	What it does
The `AstraSystem` resource	Describes a system: its image and its groups of agents. You change a system by changing this resource.
The manager agents	Watch `AstraSystem` resources and bring the cluster in line: StatefulSets for singletons, Deployments for pools, Services, and each system’s directory, which maps agent names to addresses. They also serve the dashboard, the knowledge graph and the control plane.
`mams-infra-base`	The parent POM for your application. It brings the runtime (startup, health, draining, telemetry, message routing by name) and builds a container image with Jib, without a Docker daemon.
The `ams` agent	Runs in every pod beside your agents. It reports the pod’s health to Kubernetes and serves the registry of its agents (`/ams/agents`) as JSON-LD.
The ops agent (optional)	Detects faults, diagnoses them with an LLM, and proposes recoveries that wait for your approval.

Installing the platform

You need a Kubernetes cluster (1.29 or later), kubectl and Helm (3.14 or later), a namespace, and a registry the cluster can pull your images from. The chart is installed from a checkout of the release:

git clone https://gitlab.com/mams-ucd/mams-infrastructure.git
cd mams-infrastructure
git checkout release-1.3.1
kubectl create namespace agents
kubectl label namespace agents pod-security.kubernetes.io/enforce=restricted   # recommended: the platform meets it
helm install platform charts/mams-infra -n agents

This installs the AstraSystem resource definition (cluster-wide), and the manager agents with their Role and NetworkPolicies. To try it on your own machine, scripts/dev-cluster.sh up creates a local kind cluster with an image registry.

Reading (the dashboard, the knowledge graph) works at once. Acting (scaling, restarting, pausing agents) needs a control token from a Secret:

kubectl -n agents create secret generic mams-infra-control \
    --from-literal=token="$(openssl rand -hex 24)" --from-literal=ops="$(openssl rand -hex 24)"
helm upgrade platform charts/mams-infra -n agents --reset-then-reuse-values \
    --set control.tokenSecret.name=mams-infra-control

token is the operators’ token, and ops the ops agent’s own. The Secret is mounted as files, so tokens can be rotated without a restart.

Other options are chart values:

To	Set
send every agent’s traces, logs and metrics to an OpenTelemetry collector	`telemetry.endpoint` (or `observability.lgtm.enabled=true`, an all-in-one stack for development)
detect and diagnose faults, and chat about the deployment	`ops.enabled=true`, and `ops.llm.secret.name` for a Secret with the LLM’s API key
let the ops agent act	`ops.recovery.modes`: `approve` (the default), `auto` or `never`, per action
encrypt traffic between pods	`mesh.linkerd.enabled=true` (needs Linkerd in the cluster)
sign in with your identity provider	`dashboard.oidc.issuer` and `.clientId`, with an OIDC proxy in front of the dashboard
restrict who reaches the manager	`networkPolicies.managerFrom`

Preparing a MAMS application

Make your application’s pom.xml a child of mams-infra-base, and add the MAMS group registry, which serves the MAMS and MAMS Infrastructure artifacts:

<parent>
    <groupId>mams-ucd</groupId>
    <artifactId>mams-infra-base</artifactId>
    <version>1.3.1</version>
</parent>

<repositories>
    <repository>
        <id>mams-ucd</id>
        <url>https://gitlab.com/api/v4/groups/4436501/-/packages/maven</url>
    </repository>
</repositories>

One image holds all of the system’s agent classes; each group of agents says which class to start. The agents are ordinary MAMS agents, with a few expectations:

They set up MAMS on the group’s port (mamsPort, by default 9000), which they share with the ams agent, and use the MAMS message service so that messages travel over HTTP between pods.
They are addressed by name. The runtime resolves an agent name with the system’s directory, so the same code runs in one JVM or across many machines.
They write only to /tmp, or to their persistence volume. Pods have a read-only root filesystem and run as user 65532.
They keep no secrets in beliefs. The ams agent publishes beliefs within the cluster.

A worker from the platform’s workers example shows the shape:

agent Worker extends mams.IntegratedInbox {
    module Messaging messaging;

    types work {
        formula job(int);
        formula completed(int);
    }

    rule +!main(list args) {
        messaging.installService("mams", "mams.messaging.MAMSMessageService");
        MAMSAgent::!setup(9000);
        MAMSAgent::!created("base");
        IntegratedInbox::!created("inbox");
    }

    rule @message(request, string sender, job(int id)) {
        send(inform, sender, completed(id));
    }
}

Build and push the image with Jib:

mvn package jib:build -Dimage=registry.example.org/delivery:1.0.0

Describing a system

An AstraSystem names the image and the groups of agents:

apiVersion: mams.ucd.ie/v1
kind: AstraSystem
metadata:
  name: delivery
spec:
  image: registry.example.org/delivery:1.0.0
  groups:
    - name: coordinator          # a singleton: one agent with a stable identity
      mainClass: Coordinator
      persistence: {}            # it remembers its previous life after a restart
    - name: drivers              # a pool: interchangeable replicas behind one name
      kind: pool
      mainClass: Driver
      agentName: driver
      replicas: 3

A singleton (the default) runs one agent with a stable address.
A pool runs interchangeable replicas. A message sent to the pool’s agent name goes to one of its ready replicas.

Apply it with kubectl -n agents apply -f delivery.yaml, and follow it with kubectl -n agents get astrasystem delivery (Pending, then Ready). Change the resource and apply it again to scale, add or remove groups, or restart.

Messages between agents

Agents send messages by name, wherever the other agent runs: send(request, "driver", deliver(order)) reaches one of the drivers pool’s replicas. Replies go to the sender’s own URI, which arrives with every message, so a pool member answers the agent that asked. Each message to a pool picks a ready replica afresh, so the work spreads across the pool. A singleton’s URI uses its stable DNS name, so it stays the same when its pod is replaced.

The agents of a system can reach each other; with the default NetworkPolicies, agents of other systems in the namespace cannot.

Scaling with the load

A pool can be scaled by hand (its replicas, the dashboard or the control plane) or automatically, using a load value reported by an application agent. Automatic scaling connects two pieces: ASTRA code reports the value, and the system’s YAML configuration tells the manager which pool to scale from that value.

In the Coordinator agent’s .astra file, declare the module alongside its other modules:

module mams.infra.runtime.Load load;

Inside a rule body, report the number of orders waiting for a driver. Here waitingOrders is an integer maintained by your application; update the report whenever that number changes, including when it becomes zero:

load.report("driver", waitingOrders);

The string "driver" is the metric key. Reporting it does not itself create or scale a pool.

In delivery.yaml, replace the drivers entry under spec.groups from Describing a system with this configuration. Keep the coordinator group and the rest of the manifest:

    - name: drivers
      kind: pool
      mainClass: Driver
      agentName: driver
      autoscale:
        min: 1
        max: 10
        source: coordinator
        metric: driver
        target: 5

This is part of the AstraSystem resource, rather than Helm chart values or ASTRA source code. Apply the edited manifest:

kubectl -n agents apply -f delivery.yaml

source: coordinator names the group whose agents report the load; metric: driver matches the key passed to load.report. The manager reads that group’s reported value and adjusts the drivers pool. target: 5 means approximately five waiting orders per replica, with between one and ten replicas. For example, a report of 20 calls for four replicas; a report of zero still keeps the configured minimum of one. Scaling decisions also respect tolerance and timing controls described in the autoscaling reference.

Before a replica is removed it drains: its agents receive $lifecycle.draining() and have time to finish their work, and an agent that uses the mams.infra.runtime.Lifecycle module says when it is done with lifecycle.drained(). For spreading work over a pool, the mams-infra-tasks module provides owners and workers that lease tasks, so a task survives a worker going away.

Surviving restarts

With persistence on a singleton, its agents keep a memory of their beliefs and goals on a volume (weak recovery). After a restart the agent starts afresh, and can read its previous life with ASTRA’s Memory module:

rule $memory.restarted() {
    if (memory.believed(next(int id))) {
        -+next(id);
        console.println("Recovered: carrying on from job " + id);
    }
    memory.recovered();
}

Watching and acting

The dashboard: kubectl -n agents port-forward svc/platform-manager 8080:9000, then http://localhost:8080/dashboard. It shows every system, group, pod and agent with its health, and lets you inspect an agent’s state, intentions, recent traces, beliefs and memory. Signed in (with the control token, or through your identity provider), you can pause, resume, step or terminate agents, and approve the ops agent’s proposals.
The knowledge graph: GET /knowledge/systems/delivery, and the resources it links to, describe the deployment as linked data (JSON-LD, using Hydra and the platform’s ontologies). Agents can follow these links like any other MAMS resource, and subscribe to changes with WebSub.
The control plane: scripts and agents act through /control, with the control token. Every action is recorded in an audit log:
```
curl -X POST -H "Authorization: Bearer $TOKEN" -H "From: you@example.org" \
     http://localhost:8080/control/systems/delivery/groups/drivers/scale -d '{"replicas": 5}'
```
The actions are scale, autoscale and restart (for groups), and pause, resume, step and terminate (for agents).
The ops agent: with an LLM, the dashboard’s Ask tab answers questions about the deployment, such as what is running, why something fails or what an agent is doing. The agent looks with read-only tools before it answers, and any action it proposes waits for your approval under Incidents.

Debugging a deployment

From 1.3.0, you can debug a running system’s agents with the ASTRA debugger for VS Code: breakpoints, stepping, and the agents’ beliefs and intentions. Set ASTRA_DEBUG_PORT in the env of the group you want to debug:

    - name: drivers
      mainClass: Driver
      kind: pool
      replicas: 3
      env:
        - name: ASTRA_DEBUG_PORT
          value: "4711"

Then forward the port of one of its pods to your machine, and attach to it with an ASTRA: Attach configuration (port 4711), from the project that holds the system’s sources:

kubectl port-forward pod/<pod> 4711:4711

While the debugger holds agents, the pod carries on as before:

It is not restarted. Its liveness check stays up, although the pod’s ams agent is held too.
Messages wait. The pod stays ready, so messages still reach it, wait in the agents’ queues, and are handled when you continue. In a pool, the other replicas go on working, and about one message in three (with three replicas) goes to the held one.
The platform knows. The ams registry marks the held agents heldByDebugger, the dashboard shows them as debugging, the control plane will not pause, resume or step them, and the ops agent does not report them as paused.

Only timeouts in the agents themselves notice that another agent is being debugged. Debug a test deployment rather than one others depend on, and remove ASTRA_DEBUG_PORT afterwards: anyone who can port-forward to the pod can stop its agents.

Security

The platform is secure by default, and each setting can be turned off when a cluster needs it (see the chart README):

Restricted pods: the platform’s and the systems’ pods meet the restricted Pod Security Standard (non-root, read-only root filesystem, no privilege escalation), and the systems’ pods get no ServiceAccount token.
NetworkPolicies: a system’s pods admit only the same system’s pods and the platform’s.
Workload identity: a system’s ams agents accept pause, resume, step and terminate only from the manager, which proves who it is with a projected ServiceAccount token. Act through the control plane or the dashboard.
People: the control plane needs a bearer token for every action, or a sign-in through your identity provider (OIDC), and records who acted in its audit log. Reading is open within the cluster.
Encryption: with Linkerd, traffic between pods is encrypted and authenticated (mutual TLS) without changes to MAMS.

Examples

The platform’s examples include:

Example	Shows
`hello`	One MAMS agent in one pod
`workers`	A singleton and a pool, messages by name, autoscaling and weak recovery
`in-step`	Pools autoscaled to keep several methods in step
`knowledge-watcher`	Following the knowledge graph with WebSub
`knowledge-reasoner`	Reasoning over the knowledge graph with ASTRA rules
`faults`	Agents that misbehave, for the ops agent