Deep Dive Into Kafka Rebalancing

Consumer group rebalancing is the mechanism that lets Kafka distribute partitions across a set of cooperating consumers and recover when membership changes. It is also the single most common source of latency spikes, duplicate processing, and mysterious “the consumer stopped consuming for 45 seconds” incidents. Most people treat rebalancing as a black box that occasionally misbehaves.

This article opens the box: what triggers a rebalance, how the group coordinator protocol works, why the classic “stop-the-world” model hurts, and how cooperative rebalancing and static membership fix it. The goal is to understand the protocol well enough to tune it and stop fighting it.

What Rebalancing Actually Does

A consumer group is a set of consumers that jointly read a topic (or several topics). Each partition is owned by exactly one consumer in the group at a time. Rebalancing is the process of (re)assigning partitions to consumers so that this invariant holds after any change.

graph TB
    subgraph "Topic: events (6 partitions)"
        P0[P0]
        P1[P1]
        P2[P2]
        P3[P3]
        P4[P4]
        P5[P5]
    end

    subgraph "Consumer Group: workers"
        C1[Consumer A]
        C2[Consumer B]
        C3[Consumer C]
    end

    P0 --> C1
    P1 --> C1
    P2 --> C2
    P3 --> C2
    P4 --> C3
    P5 --> C3

    style C1 fill:#d4edda,stroke:#28a745,stroke-width:2px
    style C2 fill:#d4edda,stroke:#28a745,stroke-width:2px
    style C3 fill:#d4edda,stroke:#28a745,stroke-width:2px

When a consumer joins, leaves, or dies, or when the topic gains partitions, the assignment above is no longer valid and the group must recompute it. The number of consumers that can do useful work is bounded by the partition count: with 6 partitions you can have at most 6 active consumers; extra consumers sit idle.

What Triggers a Rebalance

Rebalances are not random. Every one of them comes from a specific event:

1. A new consumer joins the group (scaling up, or a restarted instance rejoining)
2. A consumer leaves gracefully by calling close and sending a LeaveGroup request
3. A consumer is declared dead because it missed heartbeats (crash, GC pause, network partition)
4. A consumer is too slow and exceeds max.poll.interval.ms between calls to poll
5. Topic metadata changes, most commonly a partition count increase or a subscribed topic being created

The last three are the ones that cause pain in production. Categories 3 and 4 in particular are usually the result of misconfiguration or an overloaded consumer, not a genuine membership change.

flowchart TD
    Start([Group is stable])

    Start --> E1[New consumer joins]
    Start --> E2[Consumer calls close]
    Start --> E3[Heartbeat timeout<br/>session.timeout.ms exceeded]
    Start --> E4[Processing timeout<br/>max.poll.interval.ms exceeded]
    Start --> E5[Partitions added to topic]

    E1 --> R[Rebalance triggered]
    E2 --> R
    E3 --> R
    E4 --> R
    E5 --> R

    R --> Stable([New assignment,<br/>group stable again])

    style Start fill:#d4edda,stroke:#28a745,stroke-width:2px
    style R fill:#fff3cd,stroke:#ffc107,stroke-width:2px
    style Stable fill:#d4edda,stroke:#28a745,stroke-width:2px
    style E3 fill:#f8d7da,stroke:#dc3545,stroke-width:2px
    style E4 fill:#f8d7da,stroke:#dc3545,stroke-width:2px

The Group Coordinator and Two Roles

Rebalancing is driven by two roles: the group coordinator and the group leader.

The group coordinator is a broker. Each consumer group is assigned to one broker, chosen by hashing the group id onto a partition of the internal __consumer_offsets topic; the leader of that partition is the coordinator. The coordinator tracks membership, receives heartbeats, and stores committed offsets. It does not decide the partition assignment.

The group leader is one of the consumers, picked by the coordinator (usually the first to join). The leader runs the assignment algorithm. This is a deliberate design choice: assignment logic lives in the client, so new assignment strategies can ship without upgrading brokers.

sequenceDiagram
    participant C1 as Consumer A
    participant C2 as Consumer B
    participant GC as Group Coordinator<br/>(broker)

    Note over C1,GC: FindCoordinator
    C1->>GC: Which broker coordinates group "workers"?
    GC-->>C1: Coordinator is broker 2

    Note over C1,GC: JoinGroup
    C1->>GC: JoinGroup (subscription)
    C2->>GC: JoinGroup (subscription)
    GC-->>C1: You are LEADER + full member list
    GC-->>C2: You are FOLLOWER

    Note over C1,GC: Leader computes assignment
    C1->>C1: Run assignment strategy

    Note over C1,GC: SyncGroup
    C1->>GC: SyncGroup (assignment for all members)
    C2->>GC: SyncGroup (empty)
    GC-->>C1: Your partitions: [P0, P1]
    GC-->>C2: Your partitions: [P2, P3]

    Note over C1,GC: Steady state
    C1->>GC: Heartbeat
    C2->>GC: Heartbeat

    box rgb(212, 237, 218) Consumers
    participant C1
    participant C2
    end
    box rgb(255, 243, 205) Broker
    participant GC
    end

The important takeaway: the coordinator orchestrates, but the leader (a client) decides who gets what. The SyncGroup step is how the leader broadcasts its decision back through the coordinator to every member.

The JoinGroup / SyncGroup Protocol

Every rebalance walks through the same phases:

1. JoinGroup — all members re-send their subscription to the coordinator. The coordinator collects them, picks a leader, and returns the full member list to the leader.
2. Assignment — the leader runs the configured assignor over the member list and topic metadata.
3. SyncGroup — the leader sends the computed assignment to the coordinator; the coordinator forwards each member its share. Followers send empty SyncGroup requests and simply receive their partitions.

Each rebalance also bumps the generation id, a monotonically increasing counter. Any request carrying a stale generation is rejected, which fences off zombie consumers from a previous generation that are still trying to commit offsets.

Generation 4: members {A, B, C}, assignment {A:[0,1], B:[2,3], C:[4,5]}
   -> Consumer C dies
Generation 5: members {A, B},    assignment {A:[0,1,2], B:[3,4,5]}
   -> A commit with generation 4 is now rejected: ILLEGAL_GENERATION

The Stop-the-World Problem

In the original protocol, called eager rebalancing, the JoinGroup phase begins with every consumer revoking all of its partitions. Nobody processes anything until the whole group has rejoined, the leader has computed a new assignment, and SyncGroup has completed. This is the “stop-the-world” behavior.

sequenceDiagram
    participant A as Consumer A
    participant B as Consumer B
    participant C as Consumer C (new)

    Note over A,B: Processing normally
    C->>A: (joins group, triggers rebalance)

    Note over A,C: STOP THE WORLD
    A->>A: Revoke ALL partitions
    B->>B: Revoke ALL partitions
    Note over A,C: No processing happens

    A->>A: Rejoin
    B->>B: Rejoin
    C->>C: Join
    Note over A,C: Leader computes assignment

    Note over A,C: Resume with new partitions
    A->>A: Assigned [P0, P1]
    B->>B: Assigned [P2, P3]
    C->>C: Assigned [P4, P5]

The cost of a stop-the-world rebalance is proportional to the slowest step in the group: a single consumer stuck in a long GC pause or a slow onPartitionsRevoked callback stalls everyone. In large groups (hundreds of consumers), even a routine deployment where instances restart one by one can trigger a storm of back-to-back full rebalances, each freezing the entire group.

Two features exist specifically to reduce this pain: cooperative rebalancing and static membership.

Cooperative (Incremental) Rebalancing

Cooperative rebalancing changes the core assumption: a consumer only revokes the partitions it is actually losing, and keeps processing everything else throughout the rebalance. It achieves this with two JoinGroup rounds.

flowchart TD
    S1[Round 1: JoinGroup] --> A1[Leader computes target assignment]
    A1 --> A2[Members that must give up a partition<br/>revoke ONLY that partition]
    A2 --> S2[Round 2: JoinGroup triggered]
    S2 --> A3[Leader assigns the now-free partitions<br/>to their new owners]
    A3 --> Done[Everyone keeps the partitions they never lost]

    style A2 fill:#fff3cd,stroke:#ffc107,stroke-width:2px
    style Done fill:#d4edda,stroke:#28a745,stroke-width:2px

Concretely, if consumer C joins a group where A owns [0,1,2] and B owns [3,4,5]:

• Eager: A and B revoke all six partitions, then get reassigned A:[0,1], B:[2,3], C:[4,5]. All six partitions pause.
• Cooperative: the target is A:[0,1], B:[3,4], C:[2,5]. A revokes only P2, B revokes only P5. In the second round those two freed partitions go to C. Only P2 and P5 ever pause; the other four keep flowing.

The trade-off is an extra rebalance round, but each round moves far less work and never stops the whole group. For anything beyond a handful of consumers, cooperative is the better default. In modern Kafka the CooperativeSticky assignor combines this incremental protocol with sticky assignment, so partitions also tend to stay with the same consumer across rebalances, preserving warm local state and caches.

Static Membership

Cooperative rebalancing reduces the cost of each rebalance; static membership reduces the number of them.

Normally, when a consumer restarts it gets a brand-new member id, so from the coordinator’s view an old member left and a new one joined: two rebalances for one restart. This is wasteful during rolling deployments, where every instance restarts.

Static membership gives each consumer a stable group.instance.id that survives restarts. When such a consumer disconnects, the coordinator does not immediately rebalance. It waits for the session.timeout.ms window, expecting the same instance id to come back and reclaim its old partitions. If the instance returns within the window, no rebalance happens at all.

sequenceDiagram
    participant C as Consumer<br/>(group.instance.id=worker-3)
    participant GC as Coordinator

    Note over C,GC: Rolling restart of worker-3
    C->>GC: Heartbeat (id=worker-3, member=m-88)
    Note over C: Process restarts

    Note over GC: worker-3 disconnected...<br/>but it is STATIC.<br/>Wait up to session.timeout.ms

    C->>GC: JoinGroup (id=worker-3, new member id)
    GC-->>C: Welcome back, here are your OLD partitions
    Note over C,GC: No rebalance for the rest of the group

    box rgb(212, 237, 218) Consumer
    participant C
    end
    box rgb(255, 243, 205) Broker
    participant GC
    end

The cost is a deliberate one: if a static member truly dies, the group waits the full session.timeout.ms before its partitions are reassigned. Static membership therefore pairs a longer session timeout (fewer spurious rebalances) against slower failure detection. It is ideal for stateful stream processing where restarts are frequent and reshuffling state is expensive; less so where fast failover matters more than restart efficiency.

Assignment Strategies

The leader picks partitions using a configurable assignor. The common ones:

• Range — assigns contiguous ranges per topic. Simple, but with multiple topics it tends to overload the first consumers. This is the historical default and a frequent cause of uneven load.
• RoundRobin — spreads all partitions across all consumers in turn. More even, but a rebalance can reshuffle nearly everything.
• Sticky — aims for balance while minimizing movement: it keeps existing assignments where possible and only moves what is needed to rebalance.
• CooperativeSticky — sticky balance plus the incremental (cooperative) protocol described above. This is the recommended choice for most new deployments.

graph TB
    subgraph "Range (2 topics x 3 partitions, 2 consumers)"
        RA[Consumer A<br/>T1:P0,P1  T2:P0,P1]
        RB[Consumer B<br/>T1:P2  T2:P2]
    end

    subgraph "RoundRobin (same input)"
        RRA[Consumer A<br/>T1:P0,P2  T2:P1]
        RRB[Consumer B<br/>T1:P1  T2:P0,P2]
    end

    style RA fill:#f8d7da,stroke:#dc3545,stroke-width:2px
    style RB fill:#f8d7da,stroke:#dc3545,stroke-width:2px
    style RRA fill:#d4edda,stroke:#28a745,stroke-width:2px
    style RRB fill:#d4edda,stroke:#28a745,stroke-width:2px

The Range example shows the imbalance clearly: Consumer A ends up with four partitions and Consumer B with two, purely because Range assigns each topic independently from the top.

Configuring a Consumer Group in Go

The sarama consumer group API exposes the rebalance lifecycle through the ConsumerGroupHandler interface. Setup runs after a new assignment is received but before consumption starts; Cleanup runs on revocation. These are the hooks where you commit offsets and flush state so a rebalance does not lose or double-process work.

package main

import (
    "context"
    "log"

    "github.com/IBM/sarama"
)

type handler struct{}

// Setup runs at the start of a new session, after partition assignment.
func (handler) Setup(sess sarama.ConsumerGroupSession) error {
    log.Printf("assigned: %v", sess.Claims())
    return nil
}

// Cleanup runs at the end of a session, before partitions are revoked.
// Flush any buffered work and commit here so a rebalance is clean.
func (handler) Cleanup(sess sarama.ConsumerGroupSession) error {
    sess.Commit()
    return nil
}

func (handler) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    for msg := range claim.Messages() {
        // process msg ...
        sess.MarkMessage(msg, "")
    }
    return nil
}

func main() {
    cfg := sarama.NewConfig()
    cfg.Version = sarama.V3_5_0_0

    // Use the cooperative-sticky rebalance strategy.
    cfg.Consumer.Group.Rebalance.GroupStrategies = []sarama.BalanceStrategy{
        sarama.NewBalanceStrategySticky(),
    }

    // Static membership: stable id survives restarts and avoids a rebalance.
    cfg.Consumer.Group.InstanceId = "worker-3"

    // Failure detection vs. rebalance sensitivity.
    cfg.Consumer.Group.Session.Timeout = 45_000 * 1_000_000  // 45s
    cfg.Consumer.Group.Heartbeat.Interval = 3_000 * 1_000_000 // 3s

    group, err := sarama.NewConsumerGroup(
        []string{"kafka1.example.com:9092"}, "workers", cfg)
    if err != nil {
        log.Fatal(err)
    }
    defer group.Close()

    ctx := context.Background()
    for {
        // Consume blocks until a rebalance, then returns so we can rejoin.
        if err := group.Consume(ctx, []string{"events"}, handler{}); err != nil {
            log.Printf("consume error: %v", err)
        }
        if ctx.Err() != nil {
            return
        }
    }
}

The for loop around Consume is not optional: each time a rebalance occurs, Consume returns and the loop must call it again to rejoin with the new generation. Forgetting this loop is a classic reason a consumer silently stops after the first rebalance.

Tuning the Key Timeouts

Three timeouts govern how quickly failures are detected and how easily a slow consumer trips a rebalance. Getting them right eliminates most spurious rebalances.

The relationship that matters most: heartbeats run on a background thread, but max.poll.interval.ms is about your processing loop. A consumer can heartbeat happily while taking too long to process a batch, and still get kicked out for exceeding the poll interval. If your handler occasionally does slow work (large batches, external calls), raise max.poll.interval.ms or reduce max.poll.records rather than blaming the network.

flowchart TD
    Start([Frequent rebalances])

    Start --> Q1{Consumers crashing<br/>or restarting?}
    Q1 -->|Yes| F1[Real membership change.<br/>Add static membership<br/>to reduce restart churn]
    Q1 -->|No| Q2{Processing slow or<br/>bursty per batch?}

    Q2 -->|Yes| F2[Raise max.poll.interval.ms<br/>or lower max.poll.records]
    Q2 -->|No| Q3{GC pauses or<br/>network blips?}

    Q3 -->|Yes| F3[Raise session.timeout.ms;<br/>tune the JVM/GC]
    Q3 -->|No| F4[Switch to CooperativeSticky<br/>to shrink each rebalance]

    style Start fill:#f8d7da,stroke:#dc3545,stroke-width:2px
    style F1 fill:#fff3cd,stroke:#ffc107,stroke-width:2px
    style F2 fill:#fff3cd,stroke:#ffc107,stroke-width:2px
    style F3 fill:#fff3cd,stroke:#ffc107,stroke-width:2px
    style F4 fill:#d4edda,stroke:#28a745,stroke-width:2px

Rebalancing and Delivery Semantics

Rebalances are where duplicate processing usually creeps in. If a consumer processes messages but a rebalance strips its partitions before it commits, the new owner re-reads from the last committed offset and reprocesses those messages. This is not a bug; it is the at-least-once guarantee showing its edges.

The defenses:

1. Commit in Cleanup / onPartitionsRevoked so an outgoing owner records its progress before letting go.
2. Make processing idempotent so reprocessing after a rebalance is harmless.
3. Prefer cooperative rebalancing so fewer partitions change hands and fewer records are at risk.
4. Keep commits frequent enough that the reprocessing window after a rebalance stays small.

For strict exactly-once, use Kafka transactions with the read-process-write pattern, which fence off a partition’s transactional writes across a rebalance via the generation and producer epoch.

Best Practices

1. Default to CooperativeSticky for any group larger than a couple of consumers.
2. Use static membership where restarts are frequent and state is expensive to move.
3. Size session.timeout.ms for real failures, not for tolerating slow processing; that is what max.poll.interval.ms is for.
4. Keep processing per poll bounded so a slow batch never trips the poll-interval eviction.
5. Always loop around Consume so the consumer rejoins after every rebalance.
6. Commit on revocation and make handlers idempotent to survive the inevitable reprocessing window.
7. Roll deployments slowly and lean on static membership to avoid a rebalance storm.

Conclusion

Rebalancing is Kafka’s answer to a hard problem: keep exactly one owner per partition as consumers come and go, without central coordination of the assignment itself. The coordinator orchestrates through JoinGroup and SyncGroup, but a client leader decides the layout.

The evolution of the protocol is a story of shrinking the blast radius:

• Eager rebalancing stops the world on every change.
• Cooperative rebalancing revokes only what moves and keeps the rest flowing.
• Static membership avoids rebalancing entirely for restarts within the session window.

Most rebalancing pain in production is not the protocol misbehaving. It is a slow consumer tripping max.poll.interval.ms, a session timeout tuned too tight for real GC pauses, or a rolling deploy without static membership. Understand which of your rebalances are genuine membership changes and which are self-inflicted, and the “consumer stopped for 45 seconds” incidents largely disappear.