tree dbefb650ce4d32d554a0218f4333b0c92a2858c6
parent 220cbe29911718b5b49509d3f42e8851237f6249
author lakshmi-manasa-g <mgaduputi@linkedin.com> 1607377677 -0800
committer GitHub <noreply@github.com> 1607377677 -0800
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJfzqMNCRBK7hj4Ov3rIwAAdHIIAGZKUOH/4WhFnumXFdjan4C+
 6cR2Ph1Kqvj116CkgXT/m8IrvgOoU0KA7oFLcwNLHXMg26yRENZCa3TKL7PzwFCF
 aYniaLMHSnKvv1DON6k3YYJ5+4hz8BMBDAu+Wm4/2PL18eX7qsXQYwEXTusssLWi
 LNTjittVsIQV0tsPY1C85C1muaNz7QtvRznRJNB7pfXJHH9Nx8YkgXR3BqYGmnhE
 GysHPvMeQfvte+hkYqfDzpfg6dx/Qr8qh9MQ0ORYu0EdP5Q82q6rg0Dpxue0PoMK
 KKPdmTcQqFiqSuGVNXN3TPMYSNtVVFgAoBeKXlqbQS2IzsiahprJ5exBdNFVkX8=
 =Bm2u
 -----END PGP SIGNATURE-----
 

SAMZA-2606: container orchestration for AM HA (#1448)

Feature:
Main feature is YARN AM high availability. The feature ensures that the new AM can establish connection with already running containers to avoid restarting all running containers when AM dies. This PR enables the new AM to accept the list fo already running container provided by the resource manager and launch only those containers that are part of the job model but not in the running container list.

Changes:
[1] ClientHelper - job submit to RM indicates to keep containers alive across attempts
SamzaYarnAppMasterLifecycle: new AM uses the yarnid-samza id mapping and accepts the list of running containers given by RM and builds its internal state (SamzaApplicationState and YarnAppState) correctly
[2] ContainerProcessManager - removes running containers from the needed processor list prior to placing resource requests