tree 286c9bad9affa23b886afa33d4c9d2f1e42b4951
parent c7965187a6953435b13727b764d8c9d6dbe9b210
author Jason Gustafson <jason@confluent.io> 1560369257 -0700
committer Jason Gustafson <jason@confluent.io> 1560369392 -0700

KAFKA-8483/KAFKA-8484; Ensure safe handling of producerId resets (#6883)

The idempotent producer attempts to detect spurious UNKNOWN_PRODUCER_ID errors and handle them by reassigning sequence numbers to the inflight batches. The inflight batches are tracked in a PriorityQueue. The problem is that the reassignment of sequence numbers depends on the iteration order of PriorityQueue, which does not guarantee any ordering. So this can result in sequence numbers being assigned in the wrong order.  This patch fixes the problem by using a sorted set instead of a priority queue so that the iteration order preserves the sequence order. Note that resetting sequence numbers is an exceptional case.

This patch also fixes KAFKA-8484, which can cause an IllegalStateException when the producerId is reset while there are pending produce requests inflight. The solution is to ensure that sequence numbers are only reset if the producerId of a failed batch corresponds to the current producerId.

Reviewers: Guozhang Wang <wangguoz@gmail.com>