Architecting Cloud-Native Kafka: Evolution from Layered Storage to a Diskless Future
Top Three Sources' Views and TSO Validation Conclusion
Commonly Confirmed Fact: The evolution trend of Kafka is shifting from traditional "shared stateless" designs to highly decoupled architectures to adapt to the operational and economic model changes in cloud environments.
Main Disagreements or Differences: Regarding storage trade-offs, early focus was on local block storage, while the current trend is to utilize the layered storage solution introduced by KIP-405, which divides the data layer into a low-latency local layer and a high-capacity remote (like S3) storage layer.
TSO Validation Conclusion: The three sources jointly confirm that the future of Kafka lies in the concept of an "economic operating system," requiring platform teams to establish telemetry-driven cost allocation pipelines and manage the operational overhead arising from storage and rebalancing.
Commonly Confirmed Fact
Kafka's evolution is a response to the financial realities of cloud environments. Traditionally, Kafka relied on "shared stateless" designs optimized for hardware to achieve low latency. However, migrating to the cloud makes storage costs (such as cross-availability-zone data transfer fees and expensive block storage) very high, forcing Kafka to evolve from a hardware-bound system into an "economic operating system" under strict financial control.
Main Disagreements or Differences
The article points out that in storage architecture, architects need to balance latency sensitivity and cost. The introduction of KIP-405 divides data into a local block storage layer and a remote object storage layer, requiring platform teams to decide whether to enable layered storage based on data retention periods and read patterns. Furthermore, the issue of multi-tenancy isolation presents a challenge; traditionally, one needs to choose dedicated clusters or accept weak isolation, while "virtual clusters" offer an intermediate path that balances isolation and avoids infrastructure duplication.
Background and Analysis
The shift in Kafka architecture is not just a technical upgrade but a reshaping of the economic model. The transfer of storage costs means platform teams need to shift monitoring from the "procurement" cost of infrastructure to the cost of "cloud API usage." This demands client-level visibility for platform teams to attribute potential massive bills for a single replay job.
Summary of Three Sources' Views
Storage Layering: KIP-405 decouples storage, using local block storage for low latency and object storage (like S3) for high capacity, requiring a trade-off based on data retention time and read patterns.
Rebalancing Protocol: Improvements in the next-generation rebalancing protocol significantly reduce operational interruptions caused by dynamic consumer scaling, making it more suitable for Kubernetes-native automatic scaling.
Multi-tenancy Isolation: Through the "virtual cluster" model, strict tenant boundaries can be achieved, avoiding the massive cost of maintaining a separate cluster for every team.