ClickHouse DB::Exception: Cannot parse Cap'n Proto schema

The "DB::Exception: Cannot parse Cap'n Proto schema" error indicates that ClickHouse failed to load or interpret a Cap'n Proto schema file. The CANNOT_PARSE_CAPN_PROTO_SCHEMA error code appears when using the CapnProto input/output format and the associated .capnp schema file has syntax errors, missing dependencies, or structural issues that prevent ClickHouse from understanding the data layout.

Impact

This error blocks all data operations that depend on the Cap'n Proto schema:

• INSERT and SELECT operations using the CapnProto format fail immediately, before any data is processed.
• If the schema file is shared across multiple tables or pipelines, all of them are affected simultaneously.
• The error occurs at schema parse time, so it surfaces on the first query attempt after deploying a bad schema.

Common Causes

1. Syntax errors in the .capnp file -- missing semicolons, unclosed braces, invalid type declarations, or incorrect annotation syntax.
2. Missing unique ID -- Cap'n Proto requires a unique @0x... file ID at the top of each schema file. Omitting it or using a duplicate ID causes a parsing failure.
3. Import resolution failure -- the schema imports another .capnp file that ClickHouse cannot locate in the format_schema_path directory.
4. Schema file not found -- the path in format_schema does not point to an existing, readable file.
5. Incompatible schema version -- using Cap'n Proto language features that are newer than what ClickHouse's embedded parser supports.
6. Type mapping issues -- the schema uses Cap'n Proto types (such as AnyPointer or complex generics) that ClickHouse cannot map to its column types.

Troubleshooting and Resolution Steps

1. Validate the schema with the capnp command-line tool:

   capnp compile -o- /path/to/schema.capnp
   

   Fix any errors that capnp compile reports.

2. Ensure the file has a unique ID. Every .capnp file must start with a unique identifier. Generate one with:

   capnp id
   

   Then add it to the top of your file:

   @0xabcdef1234567890;
   

3. Check the schema path configuration:

   SELECT value FROM system.settings WHERE name = 'format_schema_path';
   

   Verify that the .capnp file and any imports exist in that directory.

4. Verify the struct name in format_schema. The setting should be filename:StructName:

   SET format_schema = 'my_schema:MyStruct';
   

5. Check file permissions. Ensure the ClickHouse process can read the schema file:

   ls -la /var/lib/clickhouse/format_schemas/my_schema.capnp
   

6. Test with a minimal schema. Start with the simplest possible .capnp file to confirm basic parsing works:

   @0xabcdef1234567890;
   struct TestRecord {
     id @0 :UInt64;
     name @1 :Text;
   }
   

7. Review ClickHouse server logs for more detailed error information:

   SELECT event_time, message FROM system.text_log
   WHERE message LIKE '%capnp%' OR message LIKE '%Cap%'
   ORDER BY event_time DESC LIMIT 10;
   

Best Practices

• Always validate .capnp schema files with capnp compile before deploying them to the ClickHouse schema directory.
• Use capnp id to generate unique file IDs rather than inventing them manually.
• Keep all schema dependencies (imported files) in the format_schema_path directory.
• Version control your .capnp files and test changes in a non-production environment first.
• Document the mapping between Cap'n Proto struct fields and ClickHouse table columns.

Frequently Asked Questions

Q: Where should I place .capnp files for ClickHouse?
A: In the directory configured by format_schema_path, which defaults to /var/lib/clickhouse/format_schemas/.

Q: How do I generate a unique file ID for a .capnp file?
A: Run capnp id on the command line. It outputs a random 64-bit ID like @0xa1b2c3d4e5f67890; that you place on the first line of your schema file.

Q: What Cap'n Proto types can ClickHouse handle?
A: ClickHouse supports basic types (Bool, Int8-Int64, UInt8-UInt64, Float32, Float64, Text, Data), lists, enums, and nested structs. Complex features like generics, AnyPointer, and capabilities are not supported.

Q: Can I use Cap'n Proto for both reading and writing in ClickHouse?
A: Yes. The CapnProto format works for both INSERT (input) and SELECT ... FORMAT CapnProto (output). The same schema file is used for both directions.

Q: My schema works with capnp compile but ClickHouse rejects it. What could be wrong?
A: ClickHouse uses its own Cap'n Proto parser that may not support every language feature. Simplify the schema by removing advanced features (generics, complex annotations, AnyPointer) and test again. Check ClickHouse version notes for specific Cap'n Proto compatibility details.