The short answer: Kafka has no hard limit on the number of topics, but it has practical limits on the number of partitions across the cluster. Since every topic has at least one partition, partitions are the real constraint. With KRaft (replacing ZooKeeper), Kafka comfortably handles partition counts that would have crashed a 2018-era cluster.
This guide gives you the actual numbers, what they're bounded by, and how to scale past them.
Quick Numbers
These are not theoretical maximums - they are values that have worked in production and benchmarks:
| Mode | Practical per-broker partition count | Practical cluster total |
|---|---|---|
| ZooKeeper era (Kafka <3.0) | ~4,000 partitions | ~200,000 partitions |
| KRaft (Kafka 3.3+) | 50,000+ partitions | Millions |
| KRaft (Kafka 3.7+, optimized) | 100,000+ partitions | Tens of millions in benchmarks |
Translated to topics: with a typical 6-12 partition default, a modern KRaft cluster can host hundreds of thousands of topics on commodity hardware. The ZooKeeper-era ceiling was closer to tens of thousands of topics per cluster.
If you remember one number: KRaft scales partition counts roughly 100x compared to ZooKeeper. That's because of how each handles metadata - log-based replication in KRaft vs per-partition znode writes in ZooKeeper.
What Limits Topic Count
Topics themselves are cheap - they're metadata entries. The cost comes from their partitions and replicas. Each partition adds:
1. Metadata size
Every topic and partition lives in cluster metadata: which broker has which replicas, which is the leader, what's the ISR. In ZooKeeper, that lived in znodes; in KRaft, it lives in the __cluster_metadata topic, log-replicated to all controllers.
ZooKeeper's znode model was the limiting factor pre-3.0. Each partition was a separate znode plus per-broker znodes, and the controller had to maintain a watch on each one. Metadata operations (failover, topic creation, deletion) became O(partitions) and slow.
KRaft replaced this with a log-replicated metadata topic. Brokers and controllers tail the log incrementally. Topic creation is a single record append, not a coordinated znode write across the ensemble.
2. Broker memory
Each partition replica on a broker consumes:
- Index files in memory (a few KB per segment)
- A producer state snapshot
- File descriptors (typically 3-4 per partition: data, index, time index, transaction index)
- A reference in the replica manager's in-memory state
Rough planning: budget a few KB of memory and a few file descriptors per partition replica per broker. A broker hosting 50,000 partitions needs careful tuning of ulimit -n, heap size, and OS settings.
3. Replication overhead
Each partition has a follower fetcher thread state per replica. Brokers use a fixed pool (num.replica.fetchers) of threads to fetch from leaders, multiplexed across partitions. More partitions = more multiplexing = slightly higher tail latency on replication.
4. Controller fail-over time
When the active controller fails, the new one loads cluster metadata. In ZooKeeper, this was O(partitions) reads from znodes - tens of seconds at 100k partitions, minutes at 1M. In KRaft, the new controller already has the metadata log replicated; failover is sub-second.
This is the difference that lets KRaft scale: the painful operation in ZooKeeper-era Kafka (controller failover) is essentially free in KRaft.
What Limits Cluster Scale Today
Now that ZooKeeper's bottleneck is gone, the remaining limits are:
- Broker memory and file descriptors (per-partition overhead, see above)
- Network: many partitions = many replication streams; the broker NIC eventually saturates
- Disk: each partition's segments live on disk; broker disk capacity bounds total partition data
- Tail latency under partition churn: lots of topic creates/deletes/rebalances stress the controller's metadata log even if it scales linearly
In practice, even at "millions of partitions" scale, the cluster is usually limited by client behavior (metadata refresh storms, fetch fan-out) before it's limited by Kafka itself.
Per-Topic vs Per-Partition Limits
There's no per-topic configuration cap. Naming, however, has constraints:
- Topic names must be
[a-zA-Z0-9._-]and up to 249 characters. - Names with both
.and_are discouraged (they can collide because of internal metrics naming). __(double-underscore) prefix is reserved for internal topics (__consumer_offsets,__transaction_state).
Per-broker, a hard practical ceiling: vm.max_map_count on Linux. Kafka memory-maps index files, and each partition uses several maps. Most distributions default to 65,536, which becomes the upper bound on partition replicas per broker without tuning:
sudo sysctl -w vm.max_map_count=2097152
If you're planning a high-density broker, raise it ahead of time.
How to Plan for Many Topics
Patterns that work:
- Use KRaft. If you're still on ZooKeeper at any scale, migration is the single biggest scalability lever. Apache Kafka 4.0 removes ZooKeeper entirely.
- Right-size partitions per topic. A topic with 100 partitions costs ~100x more than a topic with 1 partition for the same data volume. Don't over-partition by default.
- Consolidate low-traffic topics. A thousand topics each at 1 message/sec is more expensive than one topic with a key-based partitioning that achieves the same isolation.
- Plan disk per broker. With many topics and modest retention, disk doesn't fill up; with large retention, it can. Use
retention.bytesto bound topic-level disk usage. - Increase
num.io.threads,num.network.threadson high-density brokers. - Tune client metadata behavior. Clients refresh metadata periodically; with many topics, that's a lot of traffic. Raise
metadata.max.age.msif topics are stable.
Patterns that don't work:
- A topic per user, customer, or tenant. This is the most common cause of "we have 200,000 topics and our cluster is unhappy." Use a single topic keyed by tenant instead. Topics are for data shapes, not data dimensions.
- Auto-creating topics in production. Typos and stale code create topics that nobody cleans up. Disable
auto.create.topics.enable. - Letting deleted topics linger.
kafka-topics.sh --deleteis asynchronous and metadata stays for a while; in some failure modes, ghost metadata accumulates. Verify deletes complete.
Common Mistakes
- Designing one topic per entity. If you'd build "one DB table per user," you wouldn't - same logic applies. One topic per event type, keyed by the entity.
- Setting replication.factor=1 to "support more topics." This trades durability for quantity. Always RF=3 in production; if you can't fit, you need more brokers.
- Ignoring
__consumer_offsetssize. With many groups and many topics, this internal topic grows. Default partition count is 50, and consumer group offset metadata accumulates. - Not budgeting controller work. Topic create/delete storms (common in CI environments using a shared cluster) hammer the controller log even on KRaft.
Monitoring Topic and Partition Count
Cluster-wide metrics worth tracking:
- Total partitions per broker - capacity planning.
- Leader partition imbalance - some brokers ending up with more leaders than others.
UnderReplicatedPartitions- any non-zero means replication can't keep up; could indicate too many partitions per broker.- Controller queue size - in KRaft, the controller's
__cluster_metadatalog lag; growth means metadata changes are outpacing replication. - File descriptor and
mmapcount per broker - linear in partition count.
Pulse tracks partition density per broker, controller load, and metadata growth across your Kafka clusters, with alerts when you're approaching practical scale limits. Start a free trial to benchmark your fleet.
Frequently Asked Questions
Q: What's the maximum number of topics in a Kafka cluster?
A: There's no hard maximum. Practically, a modern KRaft cluster can host hundreds of thousands of topics on commodity hardware. The limit is partitions, not topics, and KRaft scales partition counts to millions per cluster.
Q: How many partitions can a single Kafka broker handle?
A: Tens of thousands on KRaft with reasonable tuning, vs ~4,000 on ZooKeeper-era clusters. Benchmarks have demonstrated 100,000+ partitions per broker on KRaft 3.7+ with appropriate hardware.
Q: Does the number of topics affect Kafka performance?
A: Indirectly. Each topic adds metadata overhead. Partition count is the more direct driver of cost, because partitions consume memory, file descriptors, and replication threads. A cluster with 10 topics of 1,000 partitions performs similarly to one with 1,000 topics of 10 partitions.
Q: What's the difference between ZooKeeper and KRaft for topic scaling?
A: ZooKeeper stored cluster metadata in znodes, with the controller maintaining watches on each one. Operations were O(partitions) and slowed dramatically past 200,000-300,000 partitions cluster-wide. KRaft replaces this with a log-replicated metadata topic, making the same operations effectively O(1). Kafka 4.0 removes ZooKeeper entirely.
Q: Should I have one topic per customer in a multi-tenant Kafka setup?
A: Almost never. A topic per tenant scales poorly past a few thousand tenants and complicates access control. Instead, use a single topic per event type, keyed by tenant ID, with ACLs or stream-processing isolation. Topics are for data shapes; tenant ID is data.
Q: How do I reduce the number of topics in my cluster?
A: Consolidate by event type using a key for the dimension you were using topics for. Migrate consumers and producers incrementally with MirrorMaker 2 or a one-time copy job, then kafka-topics.sh --delete the old topics once the cutover is verified.
