ClickHouse DB::Exception: Protobuf field not repeated

The PROTOBUF_FIELD_NOT_REPEATED error is raised when a ClickHouse Array column is mapped to a Protobuf field that is not declared as repeated. In Protobuf, only repeated fields can hold multiple values, so ClickHouse cannot serialize an array into a singular field. This mismatch between the table schema and the .proto definition will block any query that reads or writes data in Protobuf format for that column.

Impact

Any INSERT or SELECT operation that uses the Protobuf format will fail for the affected table when this error is triggered. If the table is fed by a streaming source such as Kafka with Protobuf deserialization, the consumer pipeline will stall. Data will not be lost in the Kafka topic, but processing will halt until the schema discrepancy is corrected.

Common Causes

1. Missing repeated keyword -- The .proto file defines the field as a scalar (e.g., string tags = 3;) when it should be repeated string tags = 3; to match a ClickHouse Array(String) column.
2. Schema evolution mistake -- A ClickHouse column was changed from a scalar type to an Array type without updating the corresponding Protobuf schema.
3. Wrong .proto file version -- The query references an older .proto file that predates the addition of the repeated modifier.
4. Copy-paste errors -- When duplicating field definitions in .proto files, the repeated keyword was accidentally omitted.

Troubleshooting and Resolution Steps

1. Read the error message carefully: The exception will name the specific Protobuf field that is not marked as repeated. Note the field name and number.

2. Check your ClickHouse table schema:

   DESCRIBE TABLE your_table;
   

   Identify which columns use Array(...) types.

3. Inspect the .proto file:

   cat /path/to/your/schema.proto
   

   For every ClickHouse Array column, the matching Protobuf field must use the repeated keyword. For example:

   message YourMessage {
       repeated string tags = 3;  // matches Array(String)
   }
   

4. Update the .proto definition: Add the repeated keyword to the field that corresponds to the Array column:

   // Before (incorrect for Array columns)
   string tags = 3;
   
   // After (correct)
   repeated string tags = 3;
   

5. Alternatively, change the ClickHouse column: If the Protobuf schema is the source of truth and the field should be scalar, alter the ClickHouse column to a non-Array type:

   ALTER TABLE your_table MODIFY COLUMN tags String;
   

6. Verify the fix:

   SELECT * FROM your_table LIMIT 5 FORMAT Protobuf
       SETTINGS format_schema = 'your_schema:YourMessage';
   

Best Practices

• Always define repeated fields in your .proto schema when the corresponding ClickHouse column is an Array type.
• Maintain a single source of truth for schema definitions and generate or validate the other schema from it.
• Add schema compatibility checks to your CI/CD pipeline to catch mismatches before deployment.
• Use descriptive comments in your .proto file to note which ClickHouse types each field maps to.

Frequently Asked Questions

Q: Can I map a ClickHouse Array(Array(String)) to a Protobuf field?
A: Not directly. Protobuf does not support nested repeated fields. You would need to wrap the inner array in a message type, e.g., repeated StringList items = 1; where StringList is a message containing repeated string values = 1;.

Q: What about Nested columns?
A: ClickHouse Nested types are internally represented as arrays of arrays. To map them to Protobuf, define a nested message type with the sub-fields, and use repeated for the outer field.

Q: Does this error apply to Protobuf 3 map fields?
A: Protobuf map fields are syntactic sugar for a repeated message with key and value fields. ClickHouse maps Map columns to Protobuf map types. This particular error applies specifically to repeated vs. non-repeated scalar mismatches, not map fields.

Q: I updated my .proto file but still get the error. What's wrong?
A: ClickHouse caches compiled Protobuf schemas. Restart the ClickHouse server or re-run the query to pick up the updated .proto file. Also verify that the format_schema setting points to the correct file path.