Physical Server Migration: Moving ClickHouse to New Hardware

Moving ClickHouse to new hardware is fundamentally a data-directory copy: ClickHouse stores everything it needs under one path (/var/lib/clickhouse by default), so a faithful copy of that directory plus the server configuration reproduces the instance on a new host. The reliable way to copy it while the source stays live is rsync in multiple passes, with a short final pass during a maintenance window.

This guide covers the rsync-based physical move for both standalone servers and replicated nodes. It is distinct from disk-to-disk moves within one host (see Move ClickHouse Table to Another Disk) and from logical backup/restore (see clickhouse-backup). Choose this approach when you want a byte-for-byte host clone — for example, retiring old hardware or moving to a new datacenter.

When to Use rsync vs. Backup/Restore

Both can move data to new hardware, but they solve different problems.

For a managed, scriptable, object-storage-backed approach, prefer clickhouse-backup. For a one-time host move where you control both ends, rsync is simpler and faster.

What Lives Under the Data Directory

Everything ClickHouse needs is under the data path (default /var/lib/clickhouse):

• data/ — table parts (the bulk of the data)
• metadata/ — CREATE statements (.sql) for databases and tables
• store/ — Atomic-database table data referenced by UUID
• user_files/, access/ — local user files and on-disk RBAC (if used)
• uuid — a per-server identity file that matters for replication

Server configuration lives separately under /etc/clickhouse-server/ (config.xml, users.xml, and everything in config.d/ and users.d/). That must be migrated too — see ClickHouse server configuration files. If you use a non-default <path>, adjust every command below accordingly.

Step 1: Prepare the New Host

Install the same ClickHouse version as the source (mixing versions during a physical copy can break part formats and metadata). Then:

# On the new host: install matching version, then stop the service
sudo systemctl stop clickhouse-server

Leave ClickHouse stopped on the destination for the whole copy. Writing into /var/lib/clickhouse while the server is running there will corrupt the copy. Make sure the destination filesystem is large enough and mounted at the same path (or plan to adjust <path> in config).

Step 2: Initial rsync Pass (Source Stays Live)

Run the first full sync while the old server keeps serving traffic. This pass moves the bulk of the data and can run for hours on large datasets:

rsync -ravlW --delete \
  /var/lib/clickhouse/ \
  user@new_host:/var/lib/clickhouse/

The flags: -a archive mode (preserves permissions, ownership, timestamps, symlinks), -r recursive, -v verbose, -l copy symlinks as symlinks, -W whole-file transfer (faster than the delta algorithm for large immutable parts), and --delete to remove destination files no longer on the source. The trailing slash on the source path matters — it copies the directory contents, not the directory itself.

To limit bandwidth impact on a busy production node, add a throttle (the value is in KB/s):

rsync -ravlW --delete --bwlimit=100000 \
  /var/lib/clickhouse/ user@new_host:/var/lib/clickhouse/

Step 3: Incremental Passes

ClickHouse parts are immutable once written and merges produce new part directories, so rsync's incremental behavior is very efficient: repeated passes transfer only newly created or merged parts. Run the same command again, as many times as you like, while the source is still live:

# Repeat as often as needed; each pass is smaller than the last
rsync -ravlW --delete /var/lib/clickhouse/ user@new_host:/var/lib/clickhouse/

Each pass shrinks the working set that the final maintenance-window pass will need to copy, which is what keeps downtime short.

Step 4: Final Cutover Pass

This is the only step that requires downtime. Stop writes (or the whole server) on the source so the on-disk state is consistent, then run one last rsync:

# On the source host
sudo systemctl stop clickhouse-server

# Final incremental pass — small because of earlier passes
rsync -ravlW --delete /var/lib/clickhouse/ user@new_host:/var/lib/clickhouse/

Stopping the server guarantees no parts are mid-merge or mid-write during the final copy. If you only stopped writes, flush first with SYSTEM FLUSH LOGS and let in-flight merges settle, but the safest option is a full stop for the final pass.

Step 5: Fix Ownership and Server Identity

After the copy, make sure the clickhouse user owns the files on the new host, and decide what to do with the server UUID:

# On the new host
sudo chown -R clickhouse:clickhouse /var/lib/clickhouse

The /var/lib/clickhouse/uuid file identifies the server. If the new host is a straight replacement that decommissions the old one (and does not coexist on the same Keeper ensemble), you can keep the copied uuid. If both servers will run against the same ZooKeeper/Keeper ensemble — for example you are adding the new host as an extra replica — you must remove it so the new host generates a fresh identity:

# Only when source and destination share a Keeper ensemble / coexist
sudo rm /var/lib/clickhouse/uuid

Leaving a duplicate UUID on two live servers sharing one Keeper ensemble causes them to fight over the same session and corrupt replication state.

Standalone Server: Start and Verify

For a non-replicated server, the move is essentially done. Migrate /etc/clickhouse-server/ (or confirm it already matches), then start:

sudo systemctl start clickhouse-server

Verify the data is intact:

-- Row counts and part health
SELECT database, table, sum(rows) AS rows, count() AS parts
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY rows DESC;

-- Confirm no broken/detached parts appeared
SELECT database, table, reason, count()
FROM system.detached_parts
GROUP BY database, table, reason;

Compare row counts against the old server before decommissioning it.

Replicated Nodes: Recovering Replication

A ReplicatedMergeTree table coordinates through ZooKeeper/Keeper. How you handle the new host depends on whether you keep the existing ensemble.

Case A: Keep the Existing Keeper Ensemble

If you are swapping one replica for new hardware but keeping the same Keeper ensemble and the same replica path, the simplest path is to not copy the uuid/replica identity by hand — instead, let the node re-register. After starting ClickHouse on the new host with matching config, if a table comes up read-only because its ZooKeeper metadata is missing or out of sync, restore it:

-- Re-attach locally present parts and push metadata back to Keeper.
-- The table must be in read-only state for this to run.
SYSTEM RESTORE REPLICA db.my_table;

-- Or across the whole cluster:
SYSTEM RESTORE REPLICA db.my_table ON CLUSTER my_cluster;

SYSTEM RESTORE REPLICA is for the case where data parts are present locally but the ZooKeeper metadata is lost — it attaches the local parts and re-creates the replica's metadata in Keeper rather than re-downloading everything from another replica. To simply re-sync a replica's session with Keeper without a full restore, use SYSTEM RESTART REPLICA db.my_table, which compares local state against Keeper and re-queues any missing tasks.

Case B: Stand Up a Fresh Keeper Ensemble

When you migrate the whole cluster — including its coordination layer — to new hardware, bring up a clean Keeper, then rebuild the replication metadata from the copied data:

1. rsync the data directory of each node as above.
2. Start the new ZooKeeper/Keeper ensemble clean (no carried-over snapshots from the old ensemble).
3. Start ClickHouse on the new nodes. Replicated tables will come up read-only because Keeper has no metadata for them.
4. Run SYSTEM RESTORE REPLICA ... ON CLUSTER ... for each replicated table to re-register the locally present parts into the fresh Keeper.

This avoids re-replicating terabytes over the network — each node keeps the data it already has and only its metadata is rebuilt in Keeper. For the underlying replication mechanics, see ClickHouse replication and production cluster configuration.

Going Standalone to Replicated During the Move

A host move is a natural moment to convert a standalone server into a replicated one. Don't try to do both in the same rsync step — first complete the physical move and verify, then convert the engines. That conversion (and its MergeTree → ReplicatedMergeTree mechanics) is covered separately in Convert MergeTree to ReplicatedMergeTree.

Best Practices

1. Match versions exactly. Install the same ClickHouse version on the destination before copying. Upgrade only after the move is verified and the server starts cleanly.
2. Multiple passes, short final pass. The whole point of incremental rsync is to keep the maintenance window small. Run several passes while live, then a quick final pass after stopping writes.
3. Stop the server for the final copy. A consistent on-disk state is worth the few minutes of downtime; copying a live data directory risks half-written parts.
4. Fix ownership immediately. chown -R clickhouse:clickhouse after the copy, before starting — wrong ownership is the most common post-rsync startup failure.
5. Decide the UUID question up front. Remove uuid only when the new host coexists with the old on the same Keeper ensemble; otherwise keep it.
6. Migrate config, not just data. /etc/clickhouse-server/ (including config.d/ and users.d/) must come along, or the new host won't behave like the old one.
7. Verify before decommissioning. Compare system.parts row counts, check system.detached_parts, and confirm replicas are out of read-only mode before tearing down the old hardware.

Common Issues

• Server won't start / permission denied. Almost always ownership: run chown -R clickhouse:clickhouse /var/lib/clickhouse. Check /var/log/clickhouse-server/clickhouse-server.err.log for the exact path it can't access.
• Tables stuck read-only after a cluster move. Expected when Keeper has no metadata for them — run SYSTEM RESTORE REPLICA. See replication problems diagnosis.
• Two replicas fighting over Keeper. Duplicate uuid copied to a node sharing the old ensemble. Remove /var/lib/clickhouse/uuid on the new node and restart.
• rsync not found in Docker. The official ClickHouse container image does not ship rsync. Install it in the container, mount the volume into a helper pod, or copy at the host volume level instead.
• Detached parts appear after the copy. Inspect system.detached_parts; parts that were mid-merge during a non-stopped final pass may land detached. Re-running the final pass after a clean stop usually resolves it.

How Pulse Helps

A physical host move is exactly the kind of operation where a small mistake — a duplicated UUID, a missed config file, a replica left read-only — surfaces hours later as silent data divergence. Pulse continuously monitors ClickHouse replication state, read-only tables, detached parts, and per-table row counts, so a migration that didn't fully complete is caught immediately rather than during the next incident. Pulse is built and run by ClickHouse, Elasticsearch, and OpenSearch experts who have done these migrations in production, and can review your cutover plan and validate the new cluster before you decommission the old hardware.

Frequently Asked Questions

Q: Do I have to stop ClickHouse to move it with rsync?

Not for the bulk of the copy. Run several incremental passes while the source is live, then stop writes (ideally the whole server) for one short final pass. Stopping the server for the final pass is what guarantees a consistent, corruption-free copy.

Q: Can I rsync to a different ClickHouse version?

No — a physical copy reproduces on-disk part and metadata formats, which are tied to the version. Install the matching version on the destination, complete the move, verify, then upgrade. If you need to change versions during the move, use logical backup/restore instead.

Q: When do I remove the uuid file?

Only when the new host will coexist with the source on the same ZooKeeper/Keeper ensemble (for example, joining as an additional replica). For a straight one-for-one hardware replacement where the old host is decommissioned, keep the copied uuid.

Q: My replicated tables are read-only after the move — what now?

That means the local data is present but Keeper has no metadata for the replica. Run SYSTEM RESTORE REPLICA db.table (or ... ON CLUSTER cluster_name). It attaches the local parts and rebuilds the replica's metadata in Keeper without re-downloading data.

Q: Does rsync handle the table schemas too?

Yes. metadata/ and store/ under the data directory hold the CREATE statements and Atomic-database table data, so a full copy of /var/lib/clickhouse includes schemas. You still need to migrate /etc/clickhouse-server/ separately for server and user configuration.

Q: How is this different from moving a table to another disk?

That operation stays on the same host and uses storage policies to relocate parts between disks — covered in Move ClickHouse Table to Another Disk. This guide moves an entire instance to a different physical machine.