Architecture Kafka Native Cloud: From Layered Storage to a Diskless Future
Top Three Sources' Views and TSO Validation Conclusion
Commonly Confirmed Fact: The evolution trend of Kafka is shifting from the traditional "shared stateless" design to a highly decoupled architecture to adapt to the operational and economic model changes in cloud environments.
Main Disagreements or Differences: Regarding storage trade-offs, early focus was on local block storage, while the current trend is to utilize the layered storage solution introduced by KIP-405, which divides the data layer into a low-latency local layer and a high-capacity remote (like S3) storage.
TSO Validation Conclusion: The three sources jointly confirm that the future of Kafka lies in the concept of an "economic operating system," requiring platform teams to establish telemetry-driven cost allocation pipelines and manage the operational overhead arising from storage and rebalancing.
Commonly Confirmed Fact
Kafka's evolution is a response to the financial realities of cloud environments. Traditionally, Kafka relied on a "shared stateless" design optimized for hardware to achieve low latency. However, migrating to the cloud makes storage costs (such as cross-availability zone data transfer fees and expensive block storage) very high, forcing Kafka to evolve from a hardware-bound system into an "economic operating system" under strict financial control.
Main Disagreements or Differences
The article points out that in terms of storage architecture, architects need to balance latency sensitivity and cost. The introduction of KIP-405 divides data into a local block storage layer and a remote object storage layer, requiring platform teams to decide whether to enable layered storage based on data retention periods and read patterns. Furthermore, the issue of multi-tenancy isolation presents a challenge; traditionally, one had to choose a dedicated cluster or accept weak isolation, while "virtual clusters" offer an intermediate path that balances isolation and avoids infrastructure duplication.
Background and Analysis
The shift in Kafka architecture is not just a technical upgrade but a reshaping of the economic model. The transfer of storage costs means that platform teams need to monitor the cost of "cloud API usage" instead of the cost of "infrastructure procurement." This requires platform teams to have client-level visibility to attribute potential large bills for a single replay job.
Summary of Three Sources' Views
Storage Layering: KIP-405 decouples storage, using local block storage for low latency and object storage (like S3) for high capacity, requiring a trade-off based on data retention time and read patterns.
Rebalancing Protocol: Improvements in the next-generation rebalancing protocol significantly reduce operational interruptions caused by dynamic consumer scaling, making it more suitable for Kubernetes native auto-scaling.
Multi-tenancy Isolation: Through the "virtual cluster" model, strict tenant boundaries can be achieved, avoiding the huge cost of maintaining a separate cluster for every team.