Kubernetes

Deploy the Restate Server on Kubernetes.

There are two main ways to deploy the Restate Server on Kubernetes:

1. Using the Restate Operator (recommended): a Kubernetes operator that simplifies deploying and managing Restate clusters and services, with advanced features like automatic service versioning and cloud environment integration.
2. Using the Helm Chart: a more bare-bone deployment method that requires more manual operational tasks.

Restate Kubernetes Operator

We recommend running Restate with the Restate Operator. The operator simplifies deploying and managing Restate clusters and services on Kubernetes.

We recommend using the Restate Operator in combination with a managed Kubernetes provider, such as AWS EKS, Azure’s AKS, or Google’s GKE. This eases the management of compute instances and persistent volumes. See the deployment guides for AWS, GCP, and Azure.

Restate Operator GitHub

Features

• Resource types to deploy, manage, and configure Restate clusters and services
• Connect Restate deployments running on Kubernetes to Restate Cloud Environments
• Automatic service versioning and scaling down old versions
• Set up network security via NetworkPolicy
• Sign requests using private keys from Secrets or CSI Secret Store
• Online volume expansion: dynamically increase the size of persistent volumes used by Restate clusters
• For AWS EKS: Manage credentials using EKS Pod Identity and security groups using Security Groups for Pods

Custom Resource Definitions

The Restate Operator introduces Custom Resource Definitions (CRDs) to Kubernetes to deploy Restate clusters and services.

It includes the following CRDs:

• RestateCluster: defines a Restate cluster deployment, including storage, networking, and configuration options.
• RestateDeployment: defines a Restate service deployment, including container image, resource requirements, and registration options (learn more).
• RestateCloudEnvironment: defines a connection between a Kubernetes service deployment and a Restate Cloud Environment (learn more).

Operator deployment

Install the Restate Operator via Helm:

helm install restate-operator \
  oci://ghcr.io/restatedev/restate-operator-helm \
  --namespace restate-operator \
  --create-namespace

To install the operator, you need to be able to create namespaces and CRDs.

Single-node deployment

The RestateCluster CRD defines a Restate cluster. The operator watches for these objects and creates the necessary Kubernetes resources, such as StatefulSet, Service, and NetworkPolicy objects in a new namespace that matches the RestateCluster name.

restate-server.yaml

apiVersion: restate.dev/v1
kind: RestateCluster
metadata:
  name: restate-test
spec:
  compute:
    image: restatedev/restate:1.5
  storage:
    storageRequestBytes: 2147483648 # 2 GiB

Apply the manifest via kubectl:

kubectl apply -f restate-server.yaml

This will deploy the Restate cluster in its own namespace, called restate-test.

To learn more about the RestateCluster spec options, see the RestateCluster pkl definition (or the less-readable yaml version).

Follow the guide to experiment with deploying a Restate cluster and services on a local kind cluster.

Replicated cluster deployment

Deploy a multi-node Restate cluster by:

1. Setting the replicas field in the compute section
2. Providing Restate cluster configuration via the config field

restate-cluster.yaml

apiVersion: restate.dev/v1
kind: RestateCluster
metadata:
  name: restate-test
spec:
  compute:
    replicas: 3
    image: restatedev/restate:1.5
  storage:
    storageRequestBytes: 2147483648 # 2 GiB
  security:
    # this kind of annotation can be used to give your cluster an IAM role in EKS
    serviceAccountAnnotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role-that-can-read-write-to-the-bucket
  config: |
    roles = [
        "worker",
        "admin",
        "log-server",
        "http-ingress",
    ]
    auto-provision = true
    default-num-partitions = 128
    default-replication = 2

    [metadata-client]
    type = "object-store"
    path = "s3://some-bucket/metadata"
    # the same aws-* parameters as below are supported here

    [bifrost]
    default-provider = "replicated"

    [worker.snapshots]
    destination = "s3://some-bucket/snapshots"
    snapshot-interval-num-records = 10000
    # you can also provide parameters here for non-S3 stores eg:
    # aws-region = "local"
    # aws-access-key-id = "minioadmin"
    # aws-secret-access-key = "minioadmin"
    # aws-endpoint-url = "http://localhost:9000"
    # aws-allow-http = true

This example deploys a 3-node Restate cluster with snapshots and metadata stored in S3.

S3 is the only supported object store for metadata storage. Other S3-compatible stores (such as MinIO) can be used for snapshots, but not for metadata storage.

Running a distributed cluster without snapshots is not recommended for production use.

Apply the manifest:

kubectl apply -f restate-cluster.yaml

You can find more information about configuring MinIO with your Restate cluster here.

Helm Chart

For a more bare-bone deployment of a Restate cluster, you can use the Helm chart. This will require you to do more manual operational tasks, such as registration and versioning of services.

Restate Helm Chart GitHub

The Helm chart does not require anything besides standard Kubernetes functionality: the ability to deploy containers and attach volumes to them. Any changes to the deployment need to be done manually by updating the Helm chart values and redeploying.

Single-node deployment

By default, the Helm chart deploys Restate as a single-replica StatefulSet:

helm install restate oci://ghcr.io/restatedev/restate-helm --namespace restate --create-namespace

View the default values.yaml

Follow the guide to experiment with deploying Restate with Helm on a local kind cluster.

Replicated cluster deployment

For a multi-node cluster, use the provided replicated configuration:

helm install restate oci://ghcr.io/restatedev/restate-helm \
  --namespace restate \
  --create-namespace \
  -f https://raw.githubusercontent.com/restatedev/restate/main/charts/restate-helm/replicated-values.yaml

View the replicated-values.yaml

After deployment, you must manually provision the cluster:

kubectl exec -n restate -it restate-0 -- restatectl provision --replication 2 --yes

Make sure to use the replicated-values.yaml file that matches your Helm chart version.

Configuration options

Key values you can customize in your own values.yaml:

• Resources: Configure CPU/memory limits and requests

  values.yaml

  resources:
    limits:
      cpu: 1
      memory: 3Gi
    requests:
      cpu: 500m
      memory: 1Gi
  

• Storage: Set persistent volume size and storage class

  values.yaml

  storage:
    size: 64Gi
    storageClassName: fast-ssd
  

• Replica count: Number of nodes in the cluster

  replicated-values.yaml

  replicaCount: 3
  

MinIO object storage

To configure a RestateCluster to send snapshots to self-hosted S3-compatible object store like MinIO, you can point the server to your MinIO instance. For security, it’s best to create a dedicated service account with credentials scoped only to the buckets Restate needs.

To deploy MinIO on your Kubernetes cluster, follow the instructions in the MinIO documentation.

MinIO should only be used for snapshots, not for metadata storage. MinIO’s consistency model may corrupt metadata when read quorum is lost, breaking your Restate cluster. Always use the default Raft metadata store for cluster metadata.

Run your MinIO instance with at least 4 replicas to avoid data loss, as recommended by MinIO.

For more information on configuring MinIO with your Restate cluster, see here.

Load balancing

When Restate is deployed as a multi-node cluster, requests can be sent to any node running the http-ingress role. Restate automatically forwards each request to the node responsible for the target partition.

Although a load balancer is not strictly required, having a single entry point is recommended. The simplest option is a DNS record with multiple A entries for round-robin resolution across all http-ingress nodes.

On Kubernetes, you can create a Service object for this purpose. The Restate operator creates these services automatically for you.

Adding a load balancer can further improve your setup by performing health checks and routing traffic only to ready nodes.

The specific Kubernetes provider to which you deploy will likely support binding to specific native load-balancer services, for example, AWS ELB in AWS EKS.