Data
Apache Kafka
Durable, replayable event streaming, built by engineers who run what they ship.
Overview
Apache Kafka is a distributed event-streaming platform. The mental model that matters is this: Kafka is not a message queue, it is a durable, append-only commit log. Producers append events to the end of a topic, those events are written to disk and kept for a configured retention period, and any number of consumers read forward through the log at their own pace. A traditional queue deletes a message once it has been consumed; Kafka keeps it, which means events can be replayed — a new service can be added months later and read the entire history from the beginning, or a consumer that hit a bug can rewind and reprocess. That single difference, retention and replay versus consume-once, is why Kafka is the backbone of so many event-driven systems and why reaching for it when you only needed a queue is such a common and expensive mistake.
Under the hood, a topic is split into partitions, and partitions are what give Kafka both its parallelism and its scale. Each partition is an ordered log that lives on one or more brokers; consumers within a consumer group divide the partitions between them, so throughput grows by adding partitions and consumers rather than by making one machine faster. Ordering is guaranteed within a partition but not across a topic, so the choice of partition key — which decides how events are distributed — quietly determines your ordering guarantees, your parallelism, and whether one hot key can bottleneck the whole stream. Getting partitioning, keys and consumer-group semantics right is most of the real engineering in a Kafka system, and it is the part that goes wrong when Kafka is dropped in without the expertise to run it.
Kafka is rarely deployed alone. The surrounding ecosystem is a large part of why teams choose it: Kafka Connect moves data in and out of databases, object stores and search indexes without bespoke plumbing; Kafka Streams and ksqlDB let you transform, join and aggregate streams in flight; and Schema Registry enforces the contract between producers and consumers so a change on one side does not silently break the other. Modern Kafka has also replaced its ZooKeeper dependency with the built-in KRaft consensus protocol, which simplifies operating a cluster considerably. We use these pieces where they earn their place, and we are equally willing to run a lean cluster with none of them when the problem does not call for them. We also use the managed options — Confluent Cloud, AWS MSK — when the goal is to run Kafka well rather than to own the operational burden of running it yourself.
Best for — High-throughput, event-driven systems where multiple independent consumers need a durable, replayable stream — and where a senior team is on hand to own partitioning, ordering and consumer-group semantics rather than hoping the defaults hold.
Why teams choose Apache Kafka
Decoupling that actually holds
Producers write events without knowing who reads them, and consumers read without coordinating with producers. New services join by subscribing to a topic, not by changing anyone else’s code. This is the difference between a system where teams can move independently and a distributed monolith held together by shared assumptions.
Replay as a first-class capability
Because Kafka retains the log rather than deleting consumed messages, you can rewind. Rebuild a broken downstream database from the event history, reprocess a stream after fixing a bug, or add a consumer that reads events from before it existed. Systems built on a retained log recover from mistakes in ways queue-based systems simply cannot.
Throughput and durability at scale
Partitioning spreads load across brokers and consumers, and replication across brokers means a node can fail without losing data or halting the stream. Kafka is engineered to sustain very high sustained write and read volumes with strong durability guarantees, which is precisely why it anchors so many real-time data platforms.
Why businesses choose Apache Kafka
- You are building an event-driven architecture where several services must consume the same events independently and stay decoupled as the system grows.
- You need durability and replay — the guarantee that no event is lost and that history can be reprocessed — not just fire-and-forget message delivery.
- Your data volume and real-time requirements have outgrown a conventional broker, and you want one pipeline feeding operational, analytical and search systems at once.
- You want a team that will tell you honestly when Kafka is the wrong answer for your scale, rather than one that reaches for it by default because it is fashionable.
What we build with Apache Kafka
The capabilities this technology is genuinely strong at — and what we most often build with it.
Topics, partitions and keys
The load-bearing decisions in any Kafka system. We design the partition count and partition key deliberately, because they set your ordering guarantees, your parallelism ceiling and whether a hot key can throttle a stream. This is the part teams skip and later pay for; we do it up front.
Consumer groups and offset management
Consumers in a group share a topic’s partitions and track their position with committed offsets. We design for the awkward cases — rebalancing when consumers join or leave, at-least-once versus exactly-once delivery, and idempotent processing — so a redeploy or a crash does not duplicate or drop events.
Kafka Connect for ingestion and egress
Rather than hand-writing integrations, we use Connect’s source and sink connectors to stream data between Kafka and databases, object stores, search indexes and warehouses — including change-data-capture from PostgreSQL or MySQL — with the reliability and back-pressure handling built in.
Stream processing with Kafka Streams and ksqlDB
For transforming, joining, windowing and aggregating events in flight, we use Kafka Streams for code-level control or ksqlDB where SQL over streams is enough. This keeps processing close to the log and stateful where it needs to be, without standing up a separate processing cluster.
Schema Registry and contracts
We enforce the producer-consumer contract with Avro, Protobuf or JSON Schema in a Schema Registry, with compatibility rules that stop a schema change on one side silently breaking the other. In an event-driven system the schema is the API, and we treat it with that seriousness.
KRaft, replication and retention
We run modern Kafka on KRaft rather than ZooKeeper, tune replication factors and in-sync-replica settings for the durability you actually need, and set retention — by time or by size, or compacted for changelog topics — to match how long events must remain replayable against what storage costs.
Use cases
Decoupling microservices
Services publish domain events to Kafka and other services react to them, replacing brittle chains of synchronous calls. Each consumer reads at its own pace, and a new service can subscribe without any existing service knowing it exists.
Real-time data pipelines
A single stream carries clickstream, telemetry or transaction events, with Connect and Streams fanning them out to analytics warehouses, search indexes and operational stores — one pipeline instead of a tangle of point-to-point integrations.
Log and event aggregation
High-volume logs, metrics and application events flow into Kafka as a durable buffer, decoupling the systems that produce them from the systems that store, index and alert on them, and absorbing spikes without dropping data.
Change-data-capture and event sourcing
Database changes are captured as an event stream, letting downstream systems stay in sync and letting you rebuild read models or caches by replaying the log — an architecture the retained, replayable log is uniquely suited to.
When Apache Kafka is the right choice
- You have multiple services that need the same stream of events, and you want them decoupled — producers should not know or care who consumes their events, and a new consumer should be able to join without touching the producer. This publish-and-forget decoupling, with each consumer group reading independently, is Kafka’s home ground.
- You genuinely need replay: the ability to reprocess history because you fixed a bug, added a feature, rebuilt a downstream store, or onboarded a new service that must see events it was not around for. A queue cannot do this; Kafka’s retained log is built for it.
- You are moving high-throughput, continuous data — clickstreams, telemetry, IoT readings, log aggregation, change-data-capture from databases — where the volume and durability requirements would overwhelm a conventional broker, and where you want a single real-time pipeline feeding analytics, search and operational systems at once.
- Wrong for: simple background jobs and task queues. If you need to send an email, resize an image, or run a deferred job, a lighter broker — RabbitMQ, Amazon SQS, or a Redis-backed queue — is the right, simpler choice, and we will tell you so. Kafka’s durability and replay are dead weight for consume-once work, and its operational cost is real.
- Wrong for: small scale with no streaming or replay need, request/response interactions, and anything you expect to query like a database. Kafka is not a database and not an RPC mechanism — if you need to look up an event by id or ask a question and wait for an answer, Kafka is the wrong tool and forcing it there is textbook over-engineering.
Apache Kafka: pros and cons
Strengths
- A durable, replayable log — not a consume-once queue — so history is available to new consumers, for reprocessing, and for rebuilding downstream state.
- Horizontal scale through partitioning: throughput grows by adding partitions, brokers and consumers rather than by scaling one machine vertically.
- A mature ecosystem — Kafka Connect, Kafka Streams, ksqlDB, Schema Registry — that covers ingestion, stream processing and contract enforcement without bespoke plumbing.
- Strong durability and ordering-within-a-partition guarantees, backed by replication, making it a dependable backbone for event-driven architectures.
Trade-offs
- Operationally heavy and genuinely hard to run well — brokers, partitions, replication, retention and consumer lag all need real attention, and self-hosting Kafka is a commitment, not a checkbox.
- It demands expertise most teams underestimate: partition-key choice, ordering guarantees, consumer-group rebalancing and exactly-once semantics are subtle, and getting them wrong produces bugs that only appear under load.
- It is frequently over-engineering. For simple queueing or background jobs, RabbitMQ, SQS or Redis do the job with a fraction of the complexity — reaching for Kafka there adds cost and moving parts you will regret.
- It is not a database and not a request/response system: you cannot query it by key or ask it a question and wait for a reply, and treating it as either leads teams into architectures that fight the tool.
Architecture
A Kafka architecture is won or lost in the topic and partition design, so that is where we start. We map the events your domain actually produces, decide what belongs on the same topic, and choose partition keys with ordering and parallelism in mind — because ordering is only guaranteed within a partition, the key that routes events is an architectural decision, not a detail. We plan replication factors and in-sync-replica requirements for the durability level the data genuinely needs, and set retention per topic: short for transient streams, long or compacted where events must remain replayable or represent current state.
Around the cluster we design how data enters and leaves. Producers are configured for the right balance of throughput, latency and durability; Kafka Connect handles the standard integrations so nobody writes bespoke ingestion code; and stream processing via Kafka Streams or ksqlDB sits where transformation belongs — close to the log. We enforce schemas through a registry so the contract between services is explicit and versioned. Where the goal is to run Kafka reliably rather than to operate it in-house, we deploy on Confluent Cloud or AWS MSK and spend the saved effort on the application instead.
Performance
Kafka is built for very high sustained throughput, and it gets there by writing sequentially to disk, batching records, and letting the operating system’s page cache serve reads — often without touching the disk at all. Performance work is therefore less about raw speed and more about tuning the levers correctly: batch size and linger for producers, fetch sizes and partition-to-consumer ratios for consumers, and compression to trade CPU for network and storage. We benchmark against your real event shapes and volumes rather than quoting headline numbers, because throughput on tiny messages and throughput on large payloads are different problems.
The failure mode we watch most closely is consumer lag — the gap between what has been produced and what has been consumed. Lag is the honest signal that consumers cannot keep up, and the fix is usually more partitions and more consumers, or faster per-message processing, not a bigger broker. We instrument lag, throughput and end-to-end latency from day one, because a Kafka system that is not measured is a Kafka system that surprises you under load.
Security
Kafka sits in the middle of your data flow, so an unsecured cluster is a serious exposure. We encrypt traffic in transit with TLS, authenticate clients — SASL, mTLS, or IAM on a managed platform — and lock down who can read and write each topic with access-control lists, so a compromised or misconfigured service cannot quietly subscribe to events it has no business seeing. On managed platforms we use the provider’s IAM and encryption-at-rest rather than reinventing them.
Beyond the cluster, we treat the schema as part of the security boundary: enforced schemas stop malformed or unexpected data entering a stream that many services trust. We are deliberate about which events carry sensitive data and for how long retention keeps them, because a long-retained topic is a long-lived copy of that data. Personal or regulated data on a stream is a decision to make consciously, with retention, access and, where needed, field-level handling designed in — not inherited from a default.
Scalability
Kafka scales horizontally by design: add partitions to raise the parallelism ceiling, add brokers to spread storage and load, and add consumers within a group to process faster. The important caveat is that partition count is easy to raise and effectively impossible to lower cleanly, and it interacts with ordering — so we size partitions for realistic growth from the start rather than discovering the ceiling in production. Replication lets the cluster tolerate broker failure without data loss, which is what makes scaling out safe rather than merely bigger.
Scaling consumers is bounded by partitions — a partition is consumed by at most one consumer in a group, so you cannot parallelise a topic beyond its partition count. We design that headroom in, and we keep processing idempotent so scaling the consumer tier up or down, and the rebalancing that comes with it, never duplicates or loses work. When storage or throughput demands shift, a well-designed cluster grows by adding nodes and partitions, not by rearchitecting the streams that run on it.
Apache Kafka integrations & ecosystem
The technologies we most often pair with it — each links to how we work with it.
How we work
We start with the events, not the cluster. Before any broker is provisioned we map the domain events your services produce and consume, agree the topic and partition design, and settle the schema and delivery-guarantee decisions — because these are cheap to get right on a whiteboard and painful to change once producers and consumers are live. If, during that work, it becomes clear you needed a queue and not a streaming platform, we say so and point you at the simpler tool rather than building you a cluster you will have to operate forever.
From there we build a real stream end to end — one producer, one topic, one consumer group, in production — and prove the guarantees under realistic load before widening. The engineers who design the topology are the ones who run it, so the choices reflect the eighteen-month operational reality, not a demo. That is what we mean by operating what we build: we tune retention, watch consumer lag, and handle rebalances because we are the ones on call when they matter.
The service behind it
Delivered throughData EngineeringWhat we build with Apache Kafka
The disciplines this technology most often shows up in — from a first build to taking over and stabilising an existing one.
How we deliver
- 01
Discover
We map the system, the constraints and the business it serves — including the parts nobody documented.
Architecture brief
- 02
Architect
Decisions get made, written down and defended before a line of production code exists.
Decision records
- 03
Build
Short cycles against working software. You see progress in the product, not in a status deck.
Shipping increments
- 04
Operate
Monitoring, incident response and iteration. The system is alive, so the engagement is too.
Runbooks & SLOs
Industries we use Apache Kafka in
Domain knowledge changes what gets built. A few of the sectors we know before the first meeting.
Also in Data
Why teams choose us for Apache Kafka
Senior engineers only
Kafka punishes guesswork — a bad partition key or a misunderstood consumer-group rebalance produces failures that only surface under load. The people designing your topology have run Kafka in production and seen how these choices age. No juniors learning on your cluster.
We operate what we build
We run the Kafka systems we ship, so we design for the on-call reality: consumer lag, rebalances, retention costs and broker failure. That means honest choices about self-hosting versus a managed platform, and topologies your own team can actually keep running.
Honest about when not to use Kafka
Kafka is the most over-reached tool in the event-driven toolbox. If you need a task queue or simple background jobs, we will point you at RabbitMQ, SQS or Redis and save you the operational weight. We would rather lose the work than build you a cluster you did not need.
Typical timeline
- 01
Event and topology design
One to two weeks mapping domain events, agreeing topic and partition design, schemas and delivery guarantees — and confirming Kafka is the right tool before anything is built.
- 02
First stream end to end
Two to three weeks delivering one producer, topic and consumer group in production, proving durability, ordering and consumer-group behaviour against realistic load.
- 03
Pipeline build-out
Adding topics, Connect integrations and stream processing incrementally, each piece shippable, with schema compatibility and consumer lag checked as we go.
- 04
Hardening and handover
Load testing, monitoring for lag and throughput, runbooks for rebalancing and broker failure, and documentation so your team can operate and extend the cluster.
How pricing works
- Fixed-scope design and build for a defined streaming need — a specific pipeline, a set of decoupled services, a change-data-capture flow — quoted once the events and guarantees are understood.
- Monthly senior engagement for ongoing event-driven platform work, where new streams and consumers are added over time and you want continuity rather than a fixed deliverable.
- Focused audits and rescues of existing Kafka deployments — partitioning, consumer lag, delivery guarantees or a cluster that is over-provisioned or misbehaving — priced by the assessment.
- Managed-platform advisory where the question is Confluent Cloud versus AWS MSK versus self-hosting, and you want an honest total-cost-of-ownership answer rather than a vendor pitch.
Hire Apache Kafka engineers
Need Apache Kafka capacity on your own team? We embed named senior engineers into your existing team — reporting to your leads, working in your rituals — so you add capacity without a hiring cycle.
Hire Apache Kafka engineersCommon questions
Is Kafka a message queue?
No, and treating it as one is the most common way teams misuse it. A queue deletes a message once it is consumed; Kafka is a durable, append-only log that retains events for a configured period, so many independent consumers can read the same events and any of them can replay history. If you only need consume-once delivery for background jobs, a real queue like RabbitMQ or SQS is simpler and the better fit.
When should we not use Kafka?
When you do not need durability, replay or high-throughput streaming across multiple consumers. For simple task queues and background jobs, RabbitMQ, SQS or a Redis-backed queue are lighter and easier to run. At small scale, or for request/response interactions, Kafka is over-engineering. And it is not a database — if you need to look up an event by id or query it, Kafka is the wrong tool.
How many partitions should a topic have?
Enough to meet your parallelism and throughput needs with room to grow, because a partition is consumed by at most one consumer in a group — partition count is your ceiling on consumer parallelism. The catch is that raising partitions is easy but lowering them cleanly is not, and the partition key controls ordering. We size partitions from realistic projected volume up front rather than discovering the limit in production.
Do we need ZooKeeper to run Kafka?
Not any more. Modern Kafka uses the built-in KRaft consensus protocol for cluster metadata, replacing the separate ZooKeeper ensemble that older deployments required. This removes a whole system to operate and simplifies running a cluster. For new deployments we run KRaft; for existing ZooKeeper-based clusters we can plan the migration when it is worth doing.
Should we self-host Kafka or use a managed service?
It depends on whether operating Kafka is a business you want to be in. Self-hosting gives full control but means owning brokers, upgrades, scaling and on-call for a genuinely demanding system. Managed options — Confluent Cloud, AWS MSK — remove most of that burden for a price. We give you an honest total-cost-of-ownership comparison for your scale rather than defaulting to either, because for many teams the managed cost is cheaper than the engineering time self-hosting consumes.
Building on Apache Kafka?
A technical conversation with the engineers who would do the work. If we are not the right fit, we will say so on the call.