ClickHouse DB::Exception: Part is permanently locked

The DB::Exception: PART_IS_LOCKED error indicates that a data part in a MergeTree-family table has been permanently locked and cannot be merged, mutated, or otherwise modified. Unlike temporary locks acquired during normal operations, a permanent lock signals that the part has been marked as unavailable, typically due to an administrative action or an internal state issue.

Impact

A permanently locked part disrupts background merge and mutation processes for the affected table. Queries that need to read the locked part may still succeed, but operations that modify or consolidate parts --- such as OPTIMIZE or ALTER UPDATE --- will fail when they encounter the lock. Over time, this can lead to an accumulation of unmerged parts.

Common Causes

1. An administrative action explicitly locked the part to prevent modifications during an investigation or backup.
2. A failed DETACH or ATTACH operation left the part in an inconsistent state with a stale lock.
3. Filesystem-level issues (such as read-only mount points) prevented ClickHouse from releasing the lock after a crash.
4. A bug in an older ClickHouse version that did not properly clean up locks during recovery.
5. Manual manipulation of data part directories on disk while the server was running.

Troubleshooting and Resolution Steps

1. Identify which parts are locked by querying the system tables:

   SELECT name, active, is_frozen
   FROM system.parts
   WHERE table = 'your_table' AND database = 'your_database';
   

2. Check the server logs for any messages about the locked part:

   grep "PART_IS_LOCKED\|locked part" /var/log/clickhouse-server/clickhouse-server.log | tail -20
   

3. If the part was frozen (e.g., by ALTER TABLE ... FREEZE), unfreeze it:

   ALTER TABLE your_database.your_table UNFREEZE WITH NAME 'backup_name';
   

4. Try detaching and re-attaching the problematic part:

   ALTER TABLE your_database.your_table DETACH PART 'part_name';
   ALTER TABLE your_database.your_table ATTACH PART 'part_name';
   

5. If the part is corrupt or unrecoverable, drop it (data loss will occur for that part):

   ALTER TABLE your_database.your_table DROP PART 'part_name';
   

6. For replicated tables, the part can often be restored from another replica after dropping:

   SYSTEM RESTORE REPLICA your_database.your_table;
   

7. Restart the ClickHouse server if lock state is suspected to be stale from a previous crash.

Best Practices

• Avoid manipulating data part directories on disk while ClickHouse is running.
• Use ALTER TABLE ... FREEZE for backups rather than manual file copying, and always unfreeze afterward.
• Monitor system.parts for parts that remain inactive or frozen for extended periods.
• Keep ClickHouse updated to benefit from fixes related to lock management.

Frequently Asked Questions

Q: Is a permanently locked part the same as a frozen part?
A: Not exactly. Freezing creates a hard link snapshot for backups and marks parts as frozen but still readable. A permanently locked part may be in a state where even merges are blocked, which goes beyond the freeze mechanism.

Q: Will a locked part cause data loss?
A: The data in the locked part is still on disk and usually readable. However, if the part cannot participate in merges, stale data may persist longer than expected in tables using ReplacingMergeTree or CollapsingMergeTree.

Q: Can I prevent parts from getting permanently locked?
A: In normal operation, permanent locks are rare. Keeping your ClickHouse version current and avoiding direct filesystem manipulation are the best preventive measures.