tree 4a38d88b16d442cfdc4e3c78cd6942aae18e0e3c
parent 1a9f356c125e2b15b69aa8c7a5f6707fb9ce3e51
author dferstay <dferstay@users.noreply.github.com> 1625060986 -0700
committer GitHub <noreply@github.com> 1625060986 +0800
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJg3HZ6CRBK7hj4Ov3rIwAAza4IAB3NIe90tk20XkOtySm/lXi3
 Gaz793VFkdzQCQEnBfC5cJOrqd62M6/s7z7/w/4DlqS7KSwD/LTctqpR7YVtufft
 D/wPwohGf7i9vdK4clvEOTtjk5PwMQW3aRRpbknYAib3lLgEcw/utlqNX9xYx4J+
 sh/bTG4sGR2L3uUAjHnPY4VyuxuieSrbyFT6IgzZ+tNQrz5C8r8pmQtqnVmPdeAH
 R0vdf/a+YZchw9wXkC98NNlUR2hnhgoX6G3jTL9Del6Hup1tH/OgdORGm01E7RNN
 ZhF2Vd6VJKNdr00bbSi63EN5mauHbv5h0HcmwrVZmPoq/mdgDL2RpkrKlUJFg+o=
 =knEn
 -----END PGP SIGNATURE-----
 

Fix panic() in internal/connection when writing to a closed channel during close (#539)

The race is as follows:
T1 - calls SendRequestNoWait(), checks the connection state, and prepares
     to enter the select statement
T2 - calls TriggerClose() closes cnx and the closeCh
T3 - run() go-routine for processing incomingRequestsCh drops into
     case <-closeCh: and calls failLeftRequestsWhenClose() which drains
     and closes incomingRequestsCh
T1 - resumes and drops into the select where both closeCh and
     incomingRequestsCh are closed.

When two cases of a `select` are valid, the case executed is chosen at
random; see https://tour.golang.org/concurrency/5

This commit introduces a connectionClosing state and a wait group to track
writes by the SendRequest() methods.
* TriggerClose() moves the connection into the connectionClosing state
  before the closeCh is closed.
* The failLeftRequestsWhenClosed() method waits on the waitgroup for
  outstanding SendRequest() methods to complete before it closes
  the incomingRequestsCh
* The SendRequest() methods first add to the waitgroup before checking the
  connection state; if the state is either closing or closed, SendRequest()
  returns an error.

With the above it is not possible for thread to attempt to add a request
to the incomingRequestsCh without being tracked by the waitgroup, and the
incomingRequestsCh will not be closed until operations tracked by the
waitgroup have completed.

Signed-off-by: Daniel Ferstay <dferstay@splunk.com>

Co-authored-by: Daniel Ferstay <dferstay@splunk.com>
Co-authored-by: xiaolongran <xiaolongran@tencent.com>