Skip to main content

Temporal Worker Controller

View Markdown

The Temporal Worker Controller is a Kubernetes controller that automates rainbow deployments of your Workers. It registers new Worker Deployment Versions, creates a Kubernetes Deployment for each version, updates the routing configuration through Temporal APIs, and removes the resources for drained versions. The Worker Controller is Generally Available, and its APIs are stable.

If you run versioned Workers on Kubernetes, the Worker Controller is the recommended way to manage rollouts and autoscaling together. You don't need the Worker Controller to use Worker Versioning, and it integrates with the Temporal CLI.

This page uses the following terms:

  • Worker Deployment: A logical service that groups similar Workers together for unified management. Each Worker Deployment has a name, such as your service name, and a series of Worker Deployment Versions.
  • Worker Deployment Version: An iteration of a Worker Deployment. Each version consists of Workers that share the same code build and environment. When a Worker starts polling for Workflow and Activity Tasks, it reports its Worker Deployment Version to the Temporal Service.
  • Deployment: A Kubernetes Deployment resource. A Deployment is "versioned" if it runs Workers that report a Worker Deployment Version.

Features​

  • Registration of new Worker Deployment Versions
  • Creation of versioned Deployment resources that manage the Pods running your Workers
  • Deletion of resources associated with drained Worker Deployment Versions
  • Manual, AllAtOnce, and Progressive rollouts of new versions
  • Automatic rollback when you set the target version back to a recent previous build
  • A "gate" Workflow that must succeed on the new version before the Worker Controller routes real traffic to it
  • Autoscaling of versioned Deployments using Kubernetes HPA or KEDA

Autoscaling versioned Workers​

The Worker Controller can manage autoscaling for versioned Worker Deployments without forcing you to choose between safe rollout behavior and elastic capacity.

Use the Worker Controller when you need all of the following:

  • Worker Versioning for safe Workflow code changes
  • Kubernetes-native rollout automation
  • autoscaling that follows each active Worker Deployment Version separately

The Worker Controller provides Kubernetes-native autoscaling through either HPA or KEDA, so you can scale on any metric available to your scaling pipeline, including:

  • CPU and memory utilization
  • Task Queue backlog metrics exposed through your metrics pipeline
  • slot utilization and other Worker-specific metrics
  • custom metrics surfaced through Prometheus or another Kubernetes metrics adapter

WorkerResourceTemplate​

To attach autoscaling or other Kubernetes resources to each Worker Deployment Version, use a WorkerResourceTemplate (WRT).

A WRT lets you define a resource template once and have the Worker Controller create a version-specific copy for each active Worker Deployment Version. This is useful for resources such as:

  • HorizontalPodAutoscaler (HPA)
  • ScaledObject (KEDA)
  • PodDisruptionBudget
  • other Kubernetes resources that should track the lifecycle of a versioned Deployment

The Worker Controller manages these resources alongside the versioned Deployments it creates, so they are updated and cleaned up as versions roll forward and drain.

By default, the Worker Controller accepts only HorizontalPodAutoscaler resources in a WorkerResourceTemplate. To use other kinds, such as PodDisruptionBudget or a KEDA ScaledObject, add them to the workerResourceTemplate.allowedResources Helm value. The webhook also checks that you have permission to create the embedded resource yourself, so a user who creates a WRT needs the matching RBAC permissions in that namespace. For details, see the WorkerResourceTemplate reference.

Choosing a scaling strategy​

The Worker Controller supports two autoscaling strategies, each attached per Worker Deployment Version through a WorkerResourceTemplate: Kubernetes HPA (with the Prometheus Adapter) and KEDA.

HPA with the Prometheus Adapter is the recommended default for most deployments. It scales independently of the number of namespaces or Task Queues and handles thousands of Task Queues efficiently. KEDA is a better fit when you need to scale from zero, have long idle periods, or require sub-minute reactivity, though it is subject to per-namespace Temporal API rate limits.

For a full comparison and a decision matrix, see Scaling recommendations in the Worker Controller repository.

Configuring Worker lifecycles​

Tag your Workers following the guidance for using Worker Versioning. You then describe each Worker Deployment as a WorkerDeployment resource. The controller creates one Kubernetes Deployment for each build.

The following example runs a Worker with a progressive rollout that is gated on the success of the HelloWorld Workflow:

apiVersion: temporal.io/v1alpha1
kind: WorkerDeployment
metadata:
  name: my-worker
spec:
  workerOptions:
    connectionRef:
      name: production-temporal
    temporalNamespace: production
  deployment:
    replicas: 3
    template:
      spec:
        containers:
          - name: worker
            image: my-worker:v2.0.0
  rollout:
    strategy: Progressive
    steps:
      - rampPercentage: 1
        pauseDuration: 30s
      - rampPercentage: 10
        pauseDuration: 1m
    gate:
      workflowType: "HelloWorld"
  sunset:
    scaledownDelay: 1h
    deleteDelay: 24h

The connectionRef points to a Connection resource that holds the Temporal Service address and either mTLS or API key credentials. A Connection uses either mTLS or an API key, not both. For Temporal Cloud, use the Namespace endpoint (<namespace>.<account>.tmprl.cloud:7233) as the hostPort. If a Namespace that also allows mTLS rejects your API key connection with a tls: certificate required error, switch hostPort to the Namespace's Regional Endpoint, such as us-east-1.aws.api.temporal.io:7233. For examples, see the Configuration reference.

When you ship a new image, the Worker Controller detects the new version and gradually makes it the Current Version of the Worker Deployment. Pinned Workflows stay on the version they started on. When older versions drain, the Worker Controller scales down their Deployments after scaledownDelay and deletes them after deleteDelay.

When you use autoscaling with the Worker Controller, each active Worker Deployment Version can scale independently while it is serving traffic. This allows older versions to drain safely while newer versions scale based on live demand.

Roll back a version​

To roll back, set the Worker image back to a previous build. If that build was the Current Version within the last hour, the Worker Controller routes 100% of traffic to it at once, regardless of the configured rollout strategy. Rollback doesn't apply to the Manual strategy, which leaves routing entirely to you.

Install the Worker Controller​

Prerequisites​

  • Kubernetes 1.19 or later
  • Helm 3.0 or later
  • Temporal Cloud, or a self-hosted Temporal Service v1.29.1 or later
  • TLS for the validating webhook. The WorkerResourceTemplate webhook is always on, so the controller Pod always needs a certificate. The recommended option is cert-manager, installed before the controller. To manage certificates yourself, see Webhook TLS.

Install the charts​

The Worker Controller ships as two Helm charts so you can upgrade the CRDs separately from the controller. Install the CRDs chart first. For the available chart versions, see the Worker Controller releases.

VERSION=<chart-version>
NAMESPACE=temporal-system

helm install temporal-worker-controller-crds \
  oci://docker.io/temporalio/temporal-worker-controller-crds \
  --version $VERSION \
  --namespace $NAMESPACE \
  --create-namespace

helm install temporal-worker-controller \
  oci://docker.io/temporalio/temporal-worker-controller \
  --version $VERSION \
  --namespace $NAMESPACE

For other deployment templates, see the Helm chart templates on GitHub.

Migrate from the previous CRD names​

In Worker Controller v1.7.0, the Worker Controller renamed its CRDs. The controller no longer reconciles resources of the old kinds, and you can't create new ones.

Old nameNew name
TemporalWorkerDeploymentWorkerDeployment
TemporalConnectionConnection
WorkerResourceTemplate.spec.temporalWorkerDeploymentRefWorkerResourceTemplate.spec.workerDeploymentRef

Keep the resource names the same, because the Worker Deployment name derives from the namespace and resource name. Don't roll back the CRDs chart after you migrate, because that can delete your Worker Deployments. For the upgrade steps, see the CRD rename migration guide.

Learn more​

The Worker Controller repository maintains the detailed guides: