kafka v0.1.0

Messaging

Apache Kafka — a distributed event-streaming platform for high-throughput publish/subscribe messaging, streams and event sourcing. Deployed as a single-node KRaft broker (no ZooKeeper) as a host-networked Nomad service.

nomad-pack run kafka --registry nomploy

…or one line (add registry + run)

curl -fsSL https://packs.nomploy.com/install.sh | sh -s -- kafka

Needs nomad-pack on PATH. The script only adds the nomploy registry and runs this pack.

Source ↗ Project ↗ ★ 33.9k ⚑ Report an issue

1 task client 9092controller 9093 1 volume image apache/kafka:3.9.1 pinned :3.9.1 image bumped today

Variables 10

values.hcl

Save as values.hcl, edit, then run:

nomad-pack run kafka -f values.hcl --registry nomploy

# The name of the Nomad job.
job_name = "kafka"

# The Nomad namespace to deploy into.
namespace = "default"

# The datacenters to deploy to.
datacenters = ["*"]

# The Apache Kafka container image. Pin a tag in production.
image = "apache/kafka:3.9.1"

# Host port for the Kafka broker (PLAINTEXT client listener).
port = 9092

# Host port for the KRaft controller listener.
controller_port = 9093

# Host/IP that clients use to reach this broker. Set to the node's address for external clients.
advertised_host = "localhost"

# Named volume for Kafka log data (/var/lib/kafka/data).
data_volume = "kafka_data"

# Placement constraints. On a nomploy cluster: attribute = "$${meta.nomploy_control_plane}", operator = "=", value = "true".
constraints = []

# The task resources.
resources = {
    cpu    = 1000
    memory = 1536
  }

Name	Type	Default	Description
job_name	string	"kafka"	The name of the Nomad job.
namespace	string	"default"	The Nomad namespace to deploy into.
datacenters	list	["*"]	The datacenters to deploy to.
image	string	"apache/kafka:3.9.1"	The Apache Kafka container image. Pin a tag in production.
port	number	9092	Host port for the Kafka broker (PLAINTEXT client listener).
controller_port	number	9093	Host port for the KRaft controller listener.
advertised_host	string	"localhost"	Host/IP that clients use to reach this broker. Set to the node's address for external clients.
data_volume	string	"kafka_data"	Named volume for Kafka log data (/var/lib/kafka/data).
constraints	list	[]	Placement constraints. On a nomploy cluster: attribute = "$${meta.nomploy_control_plane}", operator = "=", value = "true".
resources	object	{ cpu = 1000 memory = 1536 }	The task resources.

Back up this pack

This pack stores data in one Docker named volume: kafka_data

restic

# Run on the node hosting this pack. Point restic at your repo first:
#   export RESTIC_REPOSITORY="s3:https://<account>.r2.cloudflarestorage.com/<bucket>"
#   export RESTIC_PASSWORD="<repo-password>"
#   export AWS_ACCESS_KEY_ID=<key>  AWS_SECRET_ACCESS_KEY=<secret>
restic backup \
  /var/lib/docker/volumes/kafka_data/_data

rclone (sync to S3/R2)

rclone sync /var/lib/docker/volumes/kafka_data/_data backup:<bucket>/kafka_data

Paths assume the default Docker volume location (/var/lib/docker/volumes). Restore by stopping the job, restoring files into the same volume, and re-running the pack.

Readme

kafka

Apache Kafka is a distributed event-streaming platform used for high-throughput publish/subscribe messaging, stream processing, log aggregation and event sourcing. This pack runs a single-node KRaft broker (combined broker + controller, no ZooKeeper) — perfect for development, testing and small self-hosted workloads.

Deploy

nomad-pack run kafka --registry=nomploy --var advertised_host=10.0.0.5

Connect a client to <advertised_host>:9092, e.g.:

kafka-console-producer.sh --bootstrap-server <node-ip>:9092 --topic demo

Configuration

Variable	Default	Description
`image`	`apache/kafka:3.9.1`	Container image (pin a tag in production).
`port`	`9092`	Host port for the PLAINTEXT client listener.
`controller_port`	`9093`	Host port for the KRaft controller listener.
`advertised_host`	`localhost`	Address clients use to reach the broker.
`data_volume`	`kafka_data`	Volume for log data (`/var/lib/kafka/data`).
`resources`	1000 MHz / 1536 MB	CPU and memory for the task.

Important: Set advertised_host to the node's reachable IP/DNS name so external clients can connect; with the default localhost only clients on the same node work. This single-node broker uses replication factor 1 — not redundant. The image auto-formats KRaft storage on first start.