tree 80547ecb01f4b4465c259d18ff276a34495987aa
parent 1962946d2cf6804fa92c15914daa8d48a14f5e09
author mynameborat <bharath.kumarasubramanian@gmail.com> 1607535191 -0800
committer GitHub <noreply@github.com> 1607535191 -0800
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJf0QpXCRBK7hj4Ov3rIwAAdHIIAGCEfEHAPRjxtTvuT8hxpqtB
 Az8/Xlut1CPfX35rMvlNRiQwdjFLkZc/bp+xyJqP7ZBmkJTurRWSfdkvNnw1HHN/
 vTbq85+x9NwQ5ksepMMmF6Ruu0Dalfza0r9B5B3WhZa3S1KRlPmUBMWFzzzcTx5A
 YOWX2bb8by9dvVozDcBGCJG/20jwy6OnwpRze1SwyUCLKola33BFMsChpg7eo7nV
 bGq9yKHv6ozfeOAZbKIJ8FZCAytiH+W2cCaTmVUl2U5D1YUqwX3UhLXfCbvJw2Rr
 E0aHkYevbbT99SVyfmqd+gTamAdnVlKQzfV2t5f9UBPWR/alZNvHtuOVFMLH0WM=
 =qNMF
 -----END PGP SIGNATURE-----
 

SAMZA-2610: Handle Metadata changes for AM HA orchestration (#1450)

Description:
AM performs planning and job model generation for every incarnation. With AM-HA, the new job model or configuration may invalidate the containers from the previous attempt. In order to ensure correctness, we handle this by detecting these changes and restart all the containers in case of any changes to metadata (job model or configuration).

Changes:

Detect changes in metadata by reading older metadata from coordinator stream and signal the CPM
As part of resource request & orchestration, ignore the containers that are already running from the previous attempt and proceed to release them if metadata changed.
Releasing the container will signal RM through AMRM client and RM will orchestrate killing the processing container. It is different from the normal StopStreamProcessor flow as the NMClient isn't the source of truth and doesn't have context about the containers spun in the previous attempts