tree c34ee4f17a8ae2f951d27787d7cf467433bd694f
parent f857156ce7c5709be63cda0721968b1a9fe3f4bb
author Nico Kruber <nico@data-artisans.com> 1530719118 +0200
committer Nico Kruber <nico@data-artisans.com> 1530786023 +0200

[FLINK-9676][network] clarify contracts of BufferListener#notifyBufferAvailable() and fix a deadlock

When recycling exclusive buffers of a RemoteInputChannel and recycling
(other/floating) buffers to the buffer pool concurrently while the
RemoteInputChannel is registered as a listener to the buffer pool and adding the
exclusive buffer triggers a floating buffer to be recycled back to the same
buffer pool, a deadlock would occur holding locks on
LocalBufferPool#availableMemorySegments and RemoteInputChannel#bufferQueue but
acquiring them in reverse order.

One such instance would be:

Task canceler thread -> RemoteInputChannel1#releaseAllResources -> recycle floating buffers
	-> lock(LocalBufferPool#availableMemorySegments) -> RemoteInputChannel2#notifyBufferAvailable
	-> try to lock(RemoteInputChannel2#bufferQueue)

Task thread -> RemoteInputChannel2#recycle
	-> lock(RemoteInputChannel2#bufferQueue) -> bufferQueue#addExclusiveBuffer -> floatingBuffer#recycleBuffer
	-> try to lock(LocalBufferPool#availableMemorySegments)

Therefore, we decouple the listener callback from lock around
LocalBufferPool#availableMemorySegments and implicitly enforce that
RemoteInputChannel2#bufferQueue takes precedence over this lock, i.e. must
be acquired first and should never be taken after having locked on
LocalBufferPool#availableMemorySegments.

This closes #6257.