Docker Compose Hadoop Cluster

When I was learning Hadoop in March, my deployment notes had me bringing up three Ubuntu virtual machines in VMware: installing the JDK, setting up passwordless SSH, editing five or six XML files, one machine after another.

It ran, but every rebuild of the environment cost half an afternoon.

This time a different approach: pack the same three-node cluster into containers with Docker. This post covers the move from a single docker run to Compose orchestration, plus the environment-variable pitfalls and six typical failures.

Commands copy cleanly throughout, state is disposable, and when it breaks you down and start over.

Hands-on versions: the Docker ⇄ Compose converter, the DevOps Cheat Sheet, and the Env Parser.

First, Hit the Wall with One Container

What Hadoop Actually Needs from the Container

Before touching a keyboard, the accounts are worth settling.

A Hadoop cluster has a string of hard requirements on its runtime; list them and you know what the container has to make up for.

One is a stable hostname. fs.defaultFS in core-site.xml reads hdfs://master:9000, the workers file lists slave1 and slave2, and change a hostname and the whole configuration is void.

Two is SSH between nodes. Hadoop’s start and stop scripts (start-dfs.sh and friends) are essentially SSH commands fired at each node.

Three is the JDK. The JVM carries every process.

Four is network reachability. Port 9000 on the NameNode (RPC) and the DataNode report channel must be reachable in both directions.

Five is persistence. Lose the NameNode’s on-disk metadata structures (fsimage and edits) and the entire directory tree of the file system is gone.

The interesting part: one item on that list comes free in containers — clock sync.

In the VM era every machine needed NTP, or clock drift between nodes would make HDFS misread heartbeats as timeouts. All containers on one host share the kernel clock, so the clocks agree by construction.

The Comfort Zone of Pseudo-Distributed

Hadoop’s official onboarding path is pseudo-distributed: NameNode, DataNode, ResourceManager and NodeManager all running on one machine.

Into a container that is one command:

1
docker run -d --name hadoop-standalone \
2
  -h master \
3
  -p 9870:9870 -p 8088:8088 \
4
  my-hadoop:3.3.6 /usr/sbin/sshd -D

-h sets the hostname (Hadoop’s XML config references hostnames everywhere, so this step is not optional), and the port mappings expose the NameNode Web UI (9870) and the YARN page (8088).

Step into the container, run start-dfs.sh, check with jps, and the processes are all there; the HDFS page loads fine in the browser.

That is the pseudo-distributed comfort zone: half an hour to a working setup.

Multiple Nodes Are the Real Wall

But pseudo-distributed is not a cluster: no real inter-node traffic, no scenario where a DataNode drops out, and none of Hadoop’s skeleton gets learned.

For three real nodes, the most obvious move is to copy that docker run three times and change the names and the network:

1
docker network create hadoop-net
2
docker run -d --name hadoop-master -h master --net hadoop-net my-hadoop:3.3.6 /usr/sbin/sshd -D
3
docker run -d --name hadoop-slave1 -h slave1 --net hadoop-net my-hadoop:3.3.6 /usr/sbin/sshd -D
4
docker run -d --name hadoop-slave2 -h slave2 --net hadoop-net my-hadoop:3.3.6 /usr/sbin/sshd -D

The problems start showing from the second command.

First, parameter drift. Three commands maintained by hand: miss one spot when bumping the image version, forget one --net, and the three nodes are no longer on the same network. Half a day of debugging later, slave2 turns out never to have joined the custom network at all.

Second, unreproducible state. Delete and rebuild a container and the SSH host keys and the NameNode’s formatted data all reset to zero, while every config tweak typed into the container earlier is gone.

Third, ordering is managed by hand. Who starts first, who failed to come up and left hostname resolution broken for the workers file — all of it watched by eye on docker ps.

The conclusion after hitting that wall is clear: multi-container scenarios need a declarative manifest, writing down which nodes, which parameters, which network, which volumes, into one file and letting the tool reconcile.

That is exactly where Compose sits.

What Compose Maps To

Three Services, Three Nodes

Compose’s mental model is unusually blunt here: one service is one node.

The three commands above become one docker-compose.yml:

1
services:
2
  master:
3
    image: my-hadoop:3.3.6
4
    hostname: master
5
    networks: [hadoop]
6
    ports: ["127.0.0.1:9870:9870", "127.0.0.1:8088:8088"]
7
    volumes:
8
      - nn-data:/opt/hadoop/data/nn
9
    command: /usr/sbin/sshd -D
10
  slave1:
11
    image: my-hadoop:3.3.6
12
    hostname: slave1
13
    networks: [hadoop]
14
    volumes:
15
      - dn1-data:/opt/hadoop/data/dn
16
    command: /usr/sbin/sshd -D
17
  slave2:
18
    image: my-hadoop:3.3.6
19
    hostname: slave2
20
    networks: [hadoop]
21
    volumes:
22
      - dn2-data:/opt/hadoop/data/dn
23
    command: /usr/sbin/sshd -D
24
networks:
25
  hadoop:
26
volumes:
27
  nn-data:
28
  dn1-data:
29
  dn2-data:

One docker compose up -d and the three nodes come up per the manifest; parameters stop drifting.

The nodes and parameters the cluster needs are all decided by this file.

There is a detail in how the ports are written: the Web UI ports are bound with a 127.0.0.1 prefix, allowing only the host itself, since a lab cluster has no business serving admin pages to the LAN.

Inter-node RPC ports like 9000 are not mapped at all; traffic goes direct over the custom network, and exposing them would only add risk.

A Custom Network Deletes the hosts File

The most comfortable payoff here: a custom bridge network ships with its own DNS.

Containers on the same network resolve each other by service name, so master points straight at the master container’s IP.

Recall the VM-era routine: edit /etc/hosts on three machines, then keep three copies in sync whenever an IP changes. In Compose that step vanishes entirely: the workers file names the services, and the NameNode finds both DataNodes through the built-in DNS.

Restart a container onto a new IP and nothing breaks, because the name does not change.

depends_on deserves its own warning: it guarantees startup order, not readiness.

Writing depends_on: [master] only means the slave container starts after the master container does, while sshd inside master may not be listening yet. Even with a healthcheck attached, what gets checked is “the container process is alive”, not “the HDFS cluster is ready”.

So Hadoop’s startup order is not left to Compose. Let start-dfs.sh inside master pull up all the DataNodes (Hadoop was designed for centralised startup); Compose only has to boot the three machines and wire up the network.

A word on --scale: Compose can fan one service out to N instances with a single command, a blessing for stateless web services but no fit for Hadoop.

The extra instances have no hostname plan and are not in the workers file, so the NameNode does not know them.

Scaling a stateful cluster is a config change, not instance replication. The container-orchestration and big-data ecosystems do treat scaling differently.

Writing the entrypoint from Scratch

What runs in the container, and how it branches into master and slave, is decided by one entrypoint script.

The skeleton is a few dozen lines:

1
#!/bin/bash
2
set -e
3
# 1. render config templates (placeholders become real env vars)
4
render_config() {
5
  for tpl in core-site hdfs-site mapred-site yarn-site; do
6
    sed -e "s/__NAMENODE_HOST__/${NAMENODE_HOST}/g" \
7
        -e "s/__REPLICATION__/${DFS_REPLICATION}/g" \
8
        /opt/templates/$tpl.xml.tpl > $HADOOP_HOME/etc/hadoop/$tpl.xml
9
  done
10
}
11
render_config
12
# 2. start local sshd (Hadoop's start/stop scripts depend on it)
13
/usr/sbin/sshd
14
# 3. branch by role
15
if [ "$(hostname)" = "master" ]; then
16
  # format on first boot (marker file guards against formatting twice)
17
  [ -f /opt/hadoop/data/nn/FORMATTED ] || { hdfs namenode -format && touch /opt/hadoop/data/nn/FORMATTED; }
18
  start-dfs.sh && start-yarn.sh
19
else
20
  # wait for master's SSH, then hang until start-dfs.sh connects in
21
  until timeout 2 bash -c "echo > /dev/tcp/master/22" 2>/dev/null; do sleep 2; done
22
  tail -f /dev/null
23
fi

A few design decisions deserve unpacking.

Config rendering goes at the very top. However many times the container restarts, the XML is always generated from the current environment variables; changing a variable equals changing config, and it is idempotent.

Formatting needs a guard rail. Running hdfs namenode -format again over a directory that already holds data leaves the cluster with a split personality; a marker file or a non-empty data check against repeats is the number one beginner accident.

The slave branch hangs on tail -f /dev/null. A container exits when its main process exits, and a slave’s fate is “stay alive, wait for master to connect”, so an endless tail is the plainest keep-alive there is.

This entrypoint plus a dozen install lines in the Dockerfile is the whole of the image’s “magic”, under a hundred lines together.

Far easier to maintain than the operations scattered across three machines in the VM era.

Pitfalls of Environment Variable Injection

One Name, Two Faces

Environment variables caused the most trouble this time.

The first pitfall is the double meaning of ${VAR} interpolation.

image: my-hadoop:${TAG} in a Compose file interpolates on the host side. Compose reads shell environment variables and the .env file next to the compose file, and has nothing to do with the environment inside the container.

Same name, two sources: the host .env drives interpolation, the container env_file drives injection, and neither knows the other exists.

I once put the version number into the container’s env_file expecting the template to reference it; interpolation got an empty string, the image tag became my-hadoop:, and the build failed.

The fix is a layering convention: the host .env holds only what Compose itself consumes (image tag, ports, volume paths), container business variables all go through env_file, and the two sides share no names.

Interpolation syntax does offer two self-rescue rules worth noting down: ${TAG:-latest} falls back to a default when the variable is missing, while ${TAG:?err} errors out and exits. Guarding the variables that must be provided explicitly with :? moves accidents like “interpolation got an empty string” from runtime to startup.

The lesson from the other direction counts too: values in env_file are not interpolated, they are read literally, so composing one variable into another can only be done inside the entrypoint. The two sides have different syntax powers, and moving config across them is where assumptions bite hardest.

The second pitfall is precedence. When the same variable appears in both environment and env_file, environment wins.

That is a rule from the docs, yet in real projects it turns into accidents easily: a leftover debug line in environment quietly overrides carefully maintained config in env_file.

Merging and diffing the variables in both places on a schedule is the routine maintenance for this kind of project.

The third pitfall is Hadoop’s own: its configuration is all XML and knows nothing about environment variables.

fs.defaultFS in core-site.xml has to read hdfs://master:9000, and changing an environment variable inside the container does not propagate into the XML.

The answer is the template rendering from the entrypoint above: write the XML as a template with placeholders and pump the real environment variables in at startup with sed or envsubst.

Once that “env in, XML out” logic is written, resizing the cluster is only a change of environment variables.

Once .env files multiply there are handy maintenance routes too: run one through the Env Parser to get a key-value view, diff two configurations, or generate a new .env from a template — more reliable than diffing by eye.

A Short List Worth Parameterizing

A quick note on which values deserve to be pulled out into variables.

The image tag (one place to change when bumping Hadoop), DFS_REPLICATION (replica count, 3 by default on a three-node cluster; shrink to two nodes some day and it must become 2 in step, or writes keep reporting “not enough replicas”), the NameNode heap size (the -Xmx in HDFS_NAMENODE_OPTS, raise it before the file count reaches tens of millions), and the hostnames of the components.

The reverse lesson counts too: do not turn everything into a variable.

Values like the Hadoop install path, which never change over their lifetime and whose change would drag every script path along, are safer hard-coded than extracted.

The boundary for parameterizing is “there is a real scenario for changing it”; nobody dares touch the variables pulled out for symmetry’s sake afterward.

Images, Keys, and Data Volumes

One base image plus a role script, or three Dockerfiles each baking its own role?

I took the former. The three nodes differ only in hostname and startup branch; the JDK, the unpacked Hadoop and the dependency layers are identical, so building three images is pure waste. Funnelling the difference into the entrypoint’s if keeps the maintenance surface minimal.

Passwordless SSH: generate a key pair at build time and bake it into the image (public key appended to authorized_keys), then mount over the private key so it can be reused.

Doing that in a lab environment saves the wait on first boot; in production, key management is a separate serious topic and cannot be copied over.

sshd has to be running in every container, because Hadoop’s start scripts are essentially remote SSH commands.

Data volumes are the biggest win of this migration.

The NameNode metadata and the DataNode block data both sit on named volumes, so after docker compose down deletes the containers, up brings the cluster back exactly as it was. No reformatting the NameNode, no redoing the keys or the config.

The classic accident here deserves a note as well: if a volume is swapped or deleted and the NameNode formatted again after the first format, the stale clusterID left in the DataNode volume no longer matches the new NameNode, the DataNode log floods with Incompatible clusterIDs, and registration is refused.

The most frequent guest among the six typical failures; the fix is in the next section.

Also keep the two commands’ semantics apart: stop/start keep the container and only stop the process, down deletes containers but named volumes survive, down -v takes the volumes with it. The last one is “factory reset”, so think about whether the NameNode is formatted before typing it.

Memory limits belong in the manifest too: the NameNode keeps all file-system metadata on the heap, and a DataNode container grabbing memory will squeeze YARN’s NodeManager into trouble. Give master more mem_limit and the slaves less; far more dignified than guessing at logs after an OOM.

Implementing SSH trust adds one consideration over the VM era: stability of the host keys.

Every container rebuild regenerates its SSH host keys, the client’s known_hosts mismatches at once, and strict checking refuses the connection outright.

The lab shortcut is to generate one host key set at image build time (the output of ssh-keygen -A baked into a layer) and let every container share it. The lab nature of the setup underwrites the risk.

At the same time set the client’s StrictHostKeyChecking to accept-new, or just no, so every rebuild does not require cleaning known_hosts by hand.

Another difference is the granularity of trust. The VM era had three machines trusting each other; in containers it narrows to a one-way “master logs into both slaves without a password”.

Because only Hadoop’s start and stop scripts initiate SSH; slaves never connect to master on their own, so narrowing costs nothing functionally and saves maintaining two more key pairs.

Trimming permissions to the actual direction of traffic is a habit worth forming in container environments.

Six Common Failures

The high-frequency failures I hit in this build, or debugged for others, filed by symptom, cause and remedy, so next time you can match by number:

Live Nodes is 0 in the Web UI. The cause is almost certainly a clusterID mismatch (see the previous section), and the DataNode log says so outright. Remedy: stop the cluster, wipe both DataNode data volumes, reformat the NameNode, start again.
ssh: connect to host master port 22: Connection refused. sshd never came up in the master container; usually /usr/sbin/sshd was not executed in the entrypoint, or host keys are missing (ssh-keygen -A on first run). Remedy: docker compose exec master sshd -t to see the config error, then add the keys by hand.
Hostname resolution fails for the workers file. Some slave never joined the custom network (a leftover from the imperative era), or a service name is misspelled. Remedy: docker compose exec master getent hosts slave1 to verify one by one, and add the missing networks back in the yml.
The NameNode sits in safe mode and will not come out. Common with leftover volume data or a DataNode liveness rate below the threshold. Waiting out the observation period is enough. If a dev environment cannot wait, hdfs dfsadmin -safemode leave forces it out, but understand that this skips the check rather than fixing it.
No NameNode/DataNode process in jps. Nine times out of ten JAVA_HOME is not set properly in hadoop-env.sh. Hadoop does not necessarily inherit the container’s environment variables, so pointing at the JDK path explicitly in the script is safest. A JAVA_HOME is not set line in the log gives it away.
The Web UI is fine but client uploads time out. This is the most insidious one: a DataNode registers with the NameNode under its own hostname, and when a client writes a file it takes the DataNode location list from the NameNode and then connects to the DataNodes by hostname. If the host that issues the request cannot resolve those container hostnames, the write stalls at the “block replication” step. Reads (which go through NameNode metadata) work while writes (direct to DataNode) fail; that asymmetry is the fingerprint of this failure. Remedy: either add hosts mappings on the machine that initiates access, or make sure DataNodes register under resolvable names.

Wiring the Pipeline Up

For this environment to live long, two things still need solving: how the image gets built, and how changes get verified.

Image layers stack by frequency of change, lowest first: system dependencies and the JDK at the bottom (unchanged for months), the Hadoop unpack and directory layout in the middle (changes with the version), config templates and the entrypoint on top (changed daily).

That way one entrypoint edit rebuilds only the last layer, and the cache cuts a build from minutes to seconds.

The ordering of instructions in a Dockerfile is the caching strategy; get the order wrong and build speed differs by an order of magnitude.

Do not tag versions with a bare latest; a “version plus date” tag like my-hadoop:3.3.6-20260907 gives any rollback a name.

On the verification side, write a smoke test script for the cluster, five steps to a healthy environment: docker compose up -d to start the cluster; poll the master container until hdfs dfsadmin -report lists three nodes (with an overall timeout, to avoid waiting forever); hdfs dfs -put a small file and get it back to compare checksums, verifying the read-write loop; run a tiny MapReduce job or hdfs dfs -rm to clean up the traces; down to finish.

The script’s real value shows up “after a config change”: the entrypoint moved one line, a field in an XML template changed, and manual verification easily misses it while the script returns a verdict in thirty seconds.

Take it further into CI and image build plus smoke test become a fixed gate on every commit. The endpoint of lab automation is not “it runs”, it is “it runs every time”.

Data backup is the last piece of the puzzle.

The NameNode metadata in the named volume is the whole cluster’s ledger; lose it and every directory structure has to be rebuilt.

Backup takes one throwaway container command: start a temporary container with both the data volume and a host directory mounted, tar the data inside, delete it when done.

Migrating the whole environment therefore converges to three pieces of luggage: the compose file plus templates (text, into Git), the image (one tar), and the volume backup (as needed).

On any machine with Docker installed, three commands are enough to restore the entire lab cluster. It is also its most visceral win over the VM approach.

Day-to-Day Operations Checklist

The high-frequency operations once the cluster is up come down to a few lines:

docker compose up -d brings up or fills in nodes per the manifest. docker compose exec master jps shows whether each process is there. docker compose logs -f master follows the NameNode log. After editing the yml, up -d incrementally rebuilds the services that changed. stop to shut down, down to reset, down -v for scorched earth.

When debugging the network, step into a container and run getent hosts slave1 to confirm DNS; if ping fails, first check whether both sides are really on the same network.

These commands all ended up in the DevOps Cheat Sheet for on-the-spot lookup. It sorts common commands — Shell, systemd, Docker, nginx — into pages by scenario, so flipping through it while writing a manifest beats digging through official docs.

If you already have a long, tuned docker run command to migrate, the Docker ⇄ Compose converter turns it into a compose fragment to reconcile. Ports, volumes and environment variables are the three parameter classes most easily dropped in a manual translation.

One last reflection, on the boundaries of this setup: a containerised Hadoop cluster is an excellent lab and learning environment. Reproducible, disposable, config as documentation; for learning HDFS architecture, running MapReduce jobs, or practising ops commands, it is far more efficient than VMs.

But it is not a “mini production environment”: every node shares one host’s kernel and resources, one machine going down is the whole cluster going down, and real distributed failures like network partitions and node power loss cannot be simulated here at all.

To practise fault injection, you still need multiple VMs or multiple cloud hosts.

Tools each cover one stretch of road; knowing the boundary is what keeps you from mistaking a lab for a production rehearsal ground.

The two approaches’ ledgers read more clearly side by side: on startup speed, a containerised cluster takes seconds against the ten-minute order of VMs. On resource cost, three desktop-grade VMs eat a fixed tens of GB of disk and a dozen-odd GB of RAM, while an idle container cluster costs almost nothing. On snapshots and migration, one yml plus one image tar reproduces on any machine, while a VM needs a full disk export of tens of GB. On fault simulation and kernel-level experiments, VMs win outright.

Choose containers to learn architecture, choose VMs to simulate failure. That dividing line is itself more worth writing into your notes than either approach.

As for the step after the cluster is up: get hands on HDFS’s everyday commands, recorded in Hadoop Study Notes -2.

Looking back, the biggest gain from this migration was getting clear on the line between declarative and imperative.

docker run is imperative: do as much as you get through. Compose is declarative: write down the desired state, and the tool reconciles the gap.

Hadoop’s multi-node setup happens to be a perfect textbook for testing that line. It is picky about hostnames, networks, startup order and persistence alike, and any vagueness anywhere gets paid back in the logs.

Thanks for reading! Follow me if you'd like~

Docker Compose Hadoop Cluster

Mon Sep 07 2026

3667 words · 20 minutes

Notes DevOps Big Data