| Hadoop YARN Change Log |
| |
| Release 2.2.0 - UNRELEASED |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| Release 2.1.2 - UNRELEASED |
| |
| INCOMPATIBLE CHANGES |
| |
| YARN-1229. Define constraints on Auxiliary Service names. Change |
| ShuffleHandler service name from mapreduce.shuffle to |
| mapreduce_shuffle (Xuan Gong via sseth) |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| YARN-1246. Added application finish-status to ApplicationSummary for the sake |
| of testing given ApplicationHistoryServer is not yet ready. (Arpit Gupta via |
| vinodkv) |
| |
| YARN-899. Added back queue level administrator-acls so that there is no |
| regression w.r.t 1.x. (Xuan Gong via vinodkv) |
| |
| YARN-1228. Clean up Fair Scheduler configuration loading. (Sandy Ryza) |
| |
| YARN-1213. Restore config to ban submitting to undeclared pools in the |
| Fair Scheduler. (Sandy Ryza) |
| |
| YARN-1277. Added a policy based configuration for http/https in common |
| HttpServer and using the same in YARN - related to per project https config |
| support via HADOOP-10022. (Suresh Srinivas and Omkar Vinit Joshi via vinodkv) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-1128. FifoPolicy.computeShares throws NPE on empty list of Schedulables |
| (Karthik Kambatla via Sandy Ryza) |
| |
| YARN-1214. Register ClientToken MasterKey in SecretManager after it is |
| saved (Jian He via bikas) |
| |
| YARN-49. Improve distributed shell application to work on a secure cluster. |
| (Vinod Kumar Vavilapalli via hitesh) |
| |
| YARN-1157. Fixed ResourceManager UI to behave correctly when apps like |
| distributed-shell do not set tracking urls. (Xuan Gong via vinodkv) |
| |
| YARN-1221. With Fair Scheduler, reserved MB reported in RM web UI increases |
| indefinitely (Siqi Li via Sandy Ryza) |
| |
| YARN-1247. test-container-executor has gotten out of sync with the changes to |
| container-executor. (rvs via tucu) |
| |
| YARN-1070. Fixed race conditions in NodeManager during container-kill. |
| (Zhijie Shen via vinodkv) |
| |
| YARN-1215. Yarn URL should include userinfo. (Chuan Liu via cnauroth) |
| |
| YARN-1262. TestApplicationCleanup relies on all containers assigned in a |
| single heartbeat (Karthik Kambatla via Sandy Ryza) |
| |
| YARN-1260. Added webapp.http.address to yarn-default.xml so that default |
| install with https enabled doesn't have broken link on NM UI. (Omkar Vinit |
| Joshi via vinodkv) |
| |
| YARN-1141. Updating resource requests should be decoupled with updating |
| blacklist (Zhijie Shen via bikas) |
| |
| YARN-876. Node resource is added twice when node comes back from unhealthy |
| to healthy. (Peng Zhang via Sandy Ryza) |
| |
| YARN-890. Ensure CapacityScheduler doesn't round-up metric for available |
| resources. (Xuan Gong & Hitesh Shah via acmurthy) |
| |
| YARN-621. Changed YARN web app to not add paths that can cause duplicate |
| additions of authenticated filters there by causing kerberos replay errors. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-1236. FairScheduler setting queue name in RMApp is not working. |
| (Sandy Ryza) |
| |
| YARN-1256. NM silently ignores non-existent service in |
| StartContainerRequest (Xuan Gong via bikas) |
| |
| YARN-1149. NM throws InvalidStateTransitonException: Invalid event: |
| APPLICATION_LOG_HANDLING_FINISHED at RUNNING (Xuan Gong via hitesh) |
| |
| YARN-1271. "Text file busy" errors launching containers again |
| (Sandy Ryza) |
| |
| YARN-1131. $yarn logs command should return an appropriate error message if |
| YARN application is still running. (Siddharth Seth via hitesh) |
| |
| YARN-1219. FSDownload changes file suffix making FileUtil.unTar() throw |
| exception. (Shanyu Zhao via cnauroth) |
| |
| YARN-1251. TestDistributedShell#TestDSShell failed with timeout. (Xuan Gong |
| via hitesh) |
| |
| YARN-1167. Fixed Distributed Shell to not incorrectly show empty hostname |
| on RM UI. (Xuan Gong via vinodkv) |
| |
| YARN-1254. Fixed NodeManager to not pollute container's credentials. (Omkar |
| Vinit Joshi via vinodkv) |
| |
| YARN-1273. Fixed Distributed-shell to account for containers that failed |
| to start. (Hitesh Shah via vinodkv) |
| |
| YARN-1032. Fixed NPE in RackResolver. (Lohit Vijayarenu via acmurthy) |
| |
| YARN-1090. Fixed CS UI to better reflect applications as non-schedulable |
| and not as pending. (Jian He via acmurthy) |
| |
| YARN-1274. Fixed NodeManager's LinuxContainerExecutor to create user, app-dir |
| and log-dirs correctly even when there are no resources to localize for the |
| container. (Siddharth Seth via vinodkv) |
| |
| YARN-1278. Fixed NodeManager to not delete local resources for apps on resync |
| command from RM - a bug caused by YARN-1149. (Hitesh Shah via vinodkv) |
| |
| Release 2.1.1-beta - 2013-09-23 |
| |
| INCOMPATIBLE CHANGES |
| |
| YARN-707. Added user information also in the YARN ClientToken so that AMs |
| can implement authorization based on incoming users. (Jason Lowe via vinodkv) |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| YARN-589. Expose a REST API for monitoring the fair scheduler (Sandy Ryza). |
| |
| YARN-1074. Cleaned up YARN CLI application list to only display running |
| applications by default. (Xuan Gong via vinodkv) |
| |
| YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via |
| Sandy Ryza) |
| |
| YARN-942. In Fair Scheduler documentation, inconsistency on which |
| properties have prefix (Akira Ajisaka via Sandy Ryza) |
| |
| YARN-905. Add state filters to nodes CLI (Wei Yan via Sandy Ryza) |
| |
| YARN-1083. Changed ResourceManager to fail when the expiry interval is less |
| than the configured node-heartbeat interval. (Zhijie Shen via vinodkv) |
| |
| YARN-1081. Made a trivial change to YARN node CLI header to avoid potential |
| confusion. (Akira AJISAKA via vinodkv) |
| |
| YARN-1034. Remove "experimental" in the Fair Scheduler documentation. |
| (Karthik Kambatla via Sandy Ryza) |
| |
| YARN-1080. Improved help message for "yarn logs" command. (Xuan Gong via |
| vinodkv) |
| |
| YARN-771. AMRMClient support for resource blacklisting (Junping Du via |
| bikas) |
| |
| YARN-1117. Improved help messages for "yarn application" and "yarn node" |
| commands. (Xuan Gong via vinodkv) |
| |
| YARN-758. Augment MockNM to use multiple cores (Karthik Kambatla via |
| Sandy Ryza) |
| |
| YARN-1120. Made ApplicationConstants.Environment.USER definition OS neutral |
| as the corresponding value is now set correctly end-to-end. (Chuan Liu via |
| vinodkv) |
| |
| YARN-1124. Modified YARN CLI application list to display new and submitted |
| applications together with running apps by default, following up YARN-1074. |
| (Xuan Gong via vinodkv) |
| |
| YARN-1065. NM should provide AuxillaryService data to the container (Xuan |
| Gong via bikas) |
| |
| YARN-696. Changed RMWebservice apps call to take in multiple application |
| states. (Trevor Lorimer via vinodkv) |
| |
| YARN-910. Augmented auxiliary services to listen for container starts and |
| completions in addition to application events. (Alejandro Abdelnur via |
| vinodkv) |
| |
| YARN-1137. Add support whitelist for system users to Yarn |
| container-executor.c. (rvs via tucu) |
| |
| YARN-1001. Added a web-service to get statistics about per application-type |
| per state for consumption by downstream projects. (Zhijie Shen via vinodkv) |
| |
| YARN-1203. Changed YARN web-app proxy to handle http and https URLs from |
| AM registration and finish correctly. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-1204. Added separate configuration properties for https for RM and NM |
| without which servers enabled with https will also start on http ports. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-948. Changed ResourceManager to validate the release container list |
| before actually releasing them. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-966. Fixed ContainerLaunch to not fail quietly when there are no |
| localized resources due to some other failure. (Zhijie Shen via vinodkv) |
| |
| YARN-502. Fixed a state machine issue with RMNode inside ResourceManager |
| which was crashing scheduler. (Mayank Bansal via vinodkv) |
| |
| YARN-573. Shared data structures in Public Localizer and Private Localizer |
| are not Thread safe. (Omkar Vinit Joshi via jlowe) |
| |
| YARN-903. Changed ContainerManager to suppress unnecessary warnings when |
| stopping already stopped containers. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-906. Fixed a bug in NodeManager where cancelling ContainerLaunch at |
| KILLING state causes that the container to hang. (Zhijie Shen via vinodkv) |
| |
| YARN-994. HeartBeat thread in AMRMClientAsync does not handle runtime |
| exception correctly (Xuan Gong via bikas) |
| |
| YARN-337. RM handles killed application tracking URL poorly (jlowe) |
| |
| YARN-107. Fixed ResourceManager and clients to better handle |
| forceKillApplication on non-running and finished applications. (Xuan Gong |
| via vinodkv) |
| |
| YARN-643. Fixed ResourceManager to remove all tokens consistently on app |
| finish. (Xuan Gong via vinodkv) |
| |
| YARN-1006. Fixed broken rendering in the Nodes list web page on the RM web |
| UI. (Xuan Gong via vinodkv) |
| |
| YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) |
| |
| YARN-1082. Create base directories on HDFS after RM login to ensure RM |
| recovery doesn't fail in secure mode. (vinodkv via acmurthy) |
| |
| YARN-1085. Modified YARN and MR2 web-apps to do HTTP authentication in |
| secure setup with kerberos. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-1094. Fixed a blocker with RM restart code because of which RM crashes |
| when try to recover an existing app. (vinodkv) |
| |
| YARN-1008. MiniYARNCluster with multiple nodemanagers, all nodes have same |
| key for allocations. (tucu) |
| |
| YARN-602. Fixed NodeManager to not let users override some mandatory |
| environmental variables. (Kenji Kikushima via vinodkv) |
| |
| YARN-1101. Active nodes can be decremented below 0 (Robert Parker |
| via tgraves) |
| |
| YARN-981. Fixed YARN webapp so that /logs servlet works like before. (Jian He |
| via vinodkv) |
| |
| YARN-1077. Fixed TestContainerLaunch test failure on Windows. (Chuan Liu via |
| vinodkv) |
| |
| YARN-957. Fixed a bug in CapacityScheduler because of which requests that |
| need more than a node's total capability were incorrectly allocated on that |
| node causing apps to hang. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-1107. Fixed a bug in ResourceManager because of which RM in secure mode |
| fails to restart. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-1049. ContainerExistStatus should define a status for preempted |
| containers. (tucu) |
| |
| YARN-1144. Unmanaged AMs registering a tracking URI should not be |
| proxy-fied. (tucu) |
| |
| YARN-1152. Fixed a bug in ResourceManager that was causing clients to get |
| invalid client token key errors when an appliation is about to finish. |
| (Jason Lowe via vinodkv) |
| |
| YARN-292. Fixed FifoScheduler and FairScheduler to make their applications |
| data structures thread safe to avoid RM crashing with |
| ArrayIndexOutOfBoundsException. (Zhijie Shen via vinodkv) |
| |
| YARN-1025. ResourceManager and NodeManager do not load native libraries on |
| Windows. (cnauroth) |
| |
| YARN-1176. RM web services ClusterMetricsInfo total nodes doesn't include |
| unhealthy nodes (Jonathan Eagles via tgraves) |
| |
| YARN-1078. TestNodeManagerResync, TestNodeManagerShutdown, and |
| TestNodeStatusUpdater fail on Windows. (Chuan Liu via cnauroth) |
| |
| YARN-1194. TestContainerLogsPage fails with native builds (Roman Shaposhnik |
| via jlowe) |
| |
| YARN-1116. Populate AMRMTokens back to AMRMTokenSecretManager after RM |
| restarts (Jian He via bikas) |
| |
| YARN-1189. NMTokenSecretManagerInNM is not being told when applications |
| have finished (Omkar Vinit Joshi via jlowe) |
| |
| YARN-540. Race condition causing RM to potentially relaunch already |
| unregistered AMs on RM restart (Jian He via bikas) |
| |
| YARN-1184. ClassCastException during preemption enforcement. (cdouglas) |
| |
| Release 2.1.0-beta - 2013-08-22 |
| |
| INCOMPATIBLE CHANGES |
| |
| YARN-396. Rationalize AllocateResponse in RM Scheduler API. (Zhijie Shen |
| via hitesh) |
| |
| YARN-439. Flatten NodeHeartbeatResponse. (Xuan Gong via sseth) |
| |
| YARN-440. Flatten RegisterNodeManagerResponse. (Xuan Gong via sseth) |
| |
| YARN-536. Removed the unused objects ContainerStatus and ContainerStatus from |
| Container which also don't belong to the container. (Xuan Gong via vinodkv) |
| |
| YARN-486. Changed NM's startContainer API to accept Container record given by |
| RM as a direct parameter instead of as part of the ContainerLaunchContext |
| record. (Xuan Gong via vinodkv) |
| |
| YARN-444. Moved special container exit codes from YarnConfiguration to API |
| where they belong. (Sandy Ryza via vinodkv) |
| |
| YARN-441. Removed unused utility methods for collections from two API |
| records. (Xuan Gong via vinodkv) |
| |
| YARN-561. Modified NodeManager to set key information into the environment |
| of every container that it launches. (Xuan Gong via vinodkv) |
| |
| YARN-579. Stop setting the Application Token in the AppMaster env, in |
| favour of the copy present in the container token field. |
| (Vinod Kumar Vavilapalli via sseth) |
| |
| YARN-629. Make YarnRemoteException not be rooted at IOException. (Xuan Gong |
| via vinodkv) |
| |
| YARN-633. Changed RMAdminProtocol api to throw IOException and |
| YarnRemoteException. (Xuan Gong via vinodkv) |
| |
| YARN-632. Changed ContainerManager api to throw IOException and |
| YarnRemoteException. (Xuan Gong via vinodkv) |
| |
| YARN-631. Changed ClientRMProtocol api to throw IOException and |
| YarnRemoteException. (Xuan Gong via vinodkv) |
| |
| YARN-630. Changed AMRMProtocol api to throw IOException and |
| YarnRemoteException. (Xuan Gong via vinodkv) |
| |
| YARN-615. Rename ContainerLaunchContext.containerTokens to tokens. |
| (Vinod Kumar Vavilapalli via sseth) |
| |
| YARN-571. Remove user from ContainerLaunchContext. (Omkar Vinit Joshi via |
| vinodkv) |
| |
| YARN-716. Making ApplicationID immutable. (Siddharth Seth via vinodkv) |
| |
| YARN-684. ContainerManager.startContainer should use |
| ContainerTokenIdentifier instead of the entire Container. |
| (Vinod Kumar Vavilapalli via sseth) |
| |
| YARN-735. Make ApplicationAttemptId, ContaienrId and NodeId immutable. |
| (Jian He via sseth) |
| |
| YARN-749. Rename ResourceRequest.(get,set)HostName to |
| ResourceRequest.(get,set)ResourceName. (acmurthy) |
| |
| YARN-720. container-log4j.properties should not refer to mapreduce |
| property names. (Zhijie Shen via sseth) |
| |
| YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for |
| eventual retirement. (Jian He via vinodkv) |
| |
| YARN-635. Renamed YarnRemoteException to YarnException. (Siddharth Seth via |
| vinodkv) |
| |
| YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. (Bikas |
| Saha via vinodkv) |
| |
| YARN-753. Added individual factory methods for all api protocol records and |
| converted the records to be abstract classes. (Jian He via vinodkv) |
| |
| YARN-724. Moved ProtoBase from api.records to api.records.impl.pb. (Jian He |
| via vinodkv) |
| |
| YARN-759. Create Command enum in AllocateResponse (bikas) |
| |
| YARN-777. Removed unreferenced objects from .proto files. (Jian He via |
| vinodkv) |
| |
| YARN-642. Removed health parameter from ResourceManager /nodes web-service |
| and cleaned the behaviour of the status parameter. (Sandy Ryza vid vinodkv) |
| |
| YARN-530. Defined Service model strictly, implemented AbstractService for |
| robust subclassing and migrated yarn-common services. (Steve Loughran via |
| vinodkv) |
| |
| YARN-746. Renamed Service.register() and Service.unregister() to |
| registerServiceListener() & unregisterServiceListener() respectively. |
| (Steve Loughran via vinodkv) |
| |
| YARN-792. Moved NodeHealthStatus from yarn.api.record to |
| yarn.server.api.record. (Jian He via vinodkv) |
| |
| YARN-806. Moved ContainerExitStatus from yarn.api to yarn.api.records. (Jian |
| He via vinodkv) |
| |
| YARN-821. Renamed setFinishApplicationStatus to setFinalApplicationStatus in |
| FinishApplicationMasterRequest for consistency. (Jian He via vinodkv) |
| |
| YARN-787. Removed minimum resource from RegisterApplicationMasterResponse. |
| (tucu via acmurthy) |
| |
| YARN-829. Renamed RMTokenSelector to be RMDelegationTokenSelector. (Zhijie |
| Shen via vinodkv) |
| |
| YARN-828. Removed the unsed YarnVersionAnnotation. (Zhijie Shen via vinodkv) |
| |
| YARN-823. Moved RMAdmin from yarn.client to yarn.client.cli and renamed it to |
| be RMAdminCLI. (Jian He via vinodkv) |
| |
| YARN-387. Renamed YARN protocols for consistency. |
| ClientRMProtocol -> ApplicationClientProtocol |
| AMRMProtocol -> ApplicationMasterProtocol |
| ContainerManager -> ContainerManagementProtocol |
| (vinodkv via acmurthy) |
| |
| YARN-831. Removed minimum resource from GetNewApplicationResponse as a |
| follow-up to YARN-787. (Jian He via acmurthy) |
| |
| YARN-824. Added static factory methods to hadoop-yarn-client interfaces. |
| (Jian He via acmurthy) |
| |
| YARN-826. Moved Clock and SystemClock into yarn.util package. (Zhijie Shen |
| via vinodkv) |
| |
| YARN-837. Moved yarn.ClusterInfo into MapReduce project as it doesn't belong |
| to YARN. (Zhijie Shen via vinodkv) |
| |
| YARN-822. Renamed ApplicationToken to be AMRMToken, and similarly the |
| corresponding TokenSelector and SecretManager. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-610. ClientToken is no longer set in the environment of the Containers. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-834. Fixed annotations for yarn-client module, reorganized packages and |
| clearly differentiated *Async apis. (Arun C Murthy and Zhijie Shen via |
| vinodkv) |
| |
| YARN-840. Moved ProtoUtils to yarn.api.records.pb.impl. (Jian He via |
| acmurthy) |
| |
| YARN-841. Move Auxiliary service to yarn-api, annotate and document it. |
| (vinodkv) |
| |
| YARN-850. Rename getClusterAvailableResources to getAvailableResources in |
| AMRMClients (Jian He via bikas) |
| |
| YARN-694. Starting to use NMTokens to authenticate all communication with |
| NodeManagers. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-553. Replaced YarnClient.getNewApplication with |
| YarnClient.createApplication which provides a directly usable |
| ApplicationSubmissionContext to simplify the api. (Karthik Kambatla via |
| acmurthy) |
| |
| YARN-851. Share NMTokens using NMTokenCache (api-based) between AMRMClient |
| and NMClient instead of memory based approach which is used currently. (Omkar |
| Vinit Joshi via vinodkv) |
| |
| YARN-869. Move ResourceManagerAdministrationProtocol out of main YARN api. |
| (vinodkv via acmurthy) |
| |
| YARN-791. Changed RM APIs and web-services related to nodes to ensure that |
| both are consistent with each other. (Sandy Ryza via vinodkv) |
| |
| YARN-727. ClientRMProtocol.getAllApplications should accept ApplicationType as |
| a parameter. (Xuan Gong via hitesh) |
| |
| YARN-701. Use application tokens irrespective of secure or non-secure |
| mode. (vinodkv via acmurthy) |
| |
| YARN-918. Remove ApplicationAttemptId from |
| RegisterApplicationMasterRequestProto. (vinodkv via acmurthy) |
| |
| YARN-926. Modified ContainerManagerProtcol APIs to take in requests for |
| multiple containers. (Jian He via vinodkv) |
| |
| NEW FEATURES |
| |
| YARN-482. FS: Extend SchedulingMode to intermediate queues. |
| (kkambatl via tucu) |
| |
| YARN-45. Add protocol for schedulers to request containers back from |
| ApplicationMasters. (Carlo Curino, cdouglas) |
| |
| YARN-563. Add the concept of an application-type for each application. |
| (Mayank Bansal via vinodkv) |
| |
| HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows |
| Azure environments. (See breakdown of tasks below for subtasks and |
| contributors) |
| |
| YARN-422. Add a NM Client library to help application-writers. (Zhijie Shen |
| via vinodkv) |
| |
| YARN-392. Make it possible to specify hard locality constraints in resource |
| requests. (sandyr via tucu) |
| |
| YARN-326. Add multi-resource scheduling to the fair scheduler. |
| (sandyr via tucu) |
| |
| YARN-398. Make it possible to specify hard locality constraints in resource |
| requests for CapacityScheduler. (acmurthy) |
| |
| YARN-781. Exposing LOGDIR in all containers' environment which should be used |
| by containers for logging purposes. (Jian He via vinodkv) |
| |
| IMPROVEMENTS |
| |
| YARN-347. Node CLI should show CPU info besides memory in node status. |
| (Junping Du via llu) |
| |
| YARN-365. Change NM heartbeat handling to not generate a scheduler event |
| on each heartbeat. (Xuan Gong via sseth) |
| |
| YARN-380. Fix yarn node -status output to be better readable. (Omkar Vinit |
| Joshi via vinodkv) |
| |
| YARN-410. Fixed RM UI so that the new lines diagnostics for a failed app on |
| the per-application page are translated to html line breaks. (Omkar Vinit |
| Joshi via vinodkv) |
| |
| YARN-198. Added a link to RM pages from the NodeManager web app. (Jian He |
| via vinodkv) |
| |
| YARN-237. Refreshing the RM page forgets how many rows I had in my |
| Datatables (jian he via bobby) |
| |
| YARN-481. Add AM Host and RPC Port to ApplicationCLI Status Output |
| (Chris Riccomini via bikas) |
| |
| YARN-297. Improve hashCode implementations for PB records. (Xuan Gong via |
| hitesh) |
| |
| YARN-417. Create AMRMClient wrapper that provides asynchronous callbacks. |
| (Sandy Ryza via bikas) |
| |
| YARN-497. Yarn unmanaged-am launcher jar does not define a main class in |
| its manifest (Hitesh Shah via bikas) |
| |
| YARN-469. Make scheduling mode in FS pluggable. (kkambatl via tucu) |
| |
| YARN-450. Define value for * in the scheduling protocol (Zhijie Shen via |
| bikas) |
| |
| YARN-475. Remove a unused constant in the public API - |
| ApplicationConstants.AM_APP_ATTEMPT_ID_ENV. (Hitesh Shah via vinodkv) |
| |
| YARN-309. Changed NodeManager to obtain heart-beat interval from the |
| ResourceManager. (Xuan Gong via vinodkv) |
| |
| YARN-447. Move ApplicationComparator in CapacityScheduler to use comparator |
| in ApplicationId. (Nemon Lou via vinodkv) |
| |
| YARN-381. Improve fair scheduler docs. (Sandy Ryza via tomwhite) |
| |
| YARN-458. YARN daemon addresses must be placed in many different configs. |
| (sandyr via tucu) |
| |
| YARN-193. Scheduler.normalizeRequest does not account for allocation |
| requests that exceed maximumAllocation limits (Zhijie Shen via bikas) |
| |
| YARN-479. NM retry behavior for connection to RM should be similar for |
| lost heartbeats (Jian He via bikas) |
| |
| YARN-495. Changed NM reboot behaviour to be a simple resync - kill all |
| containers and re-register with RM. (Jian He via vinodkv) |
| |
| YARN-514. Delayed store operations should not result in RM unavailability |
| for app submission (Zhijie Shen via bikas) |
| |
| YARN-586. Fixed a typo in ApplicationSubmissionContext#setApplicationId. |
| (Zhijie Shen via vinodkv) |
| |
| YARN-542. Changed the default global AM max-attempts value to be not one. |
| (Zhijie Shen via vinodkv) |
| |
| YARN-583. Moved application level local resources to be localized under the |
| filecache sub-directory under application directory. (Omkar Vinit Joshi via |
| vinodkv) |
| |
| YARN-581. Added a test to verify that app delegation tokens are restored |
| after RM restart. (Jian He via vinodkv) |
| |
| YARN-577. Add application-progress also to ApplicationReport. (Hitesh Shah |
| via vinodkv) |
| |
| YARN-595. Refactor fair scheduler to use common Resources. (Sandy Ryza |
| via tomwhite) |
| |
| YARN-562. Modified NM to reject any containers allocated by a previous |
| ResourceManager. (Jian He via vinodkv) |
| |
| YARN-591. Moved RM recovery related records out of public API as they do not |
| belong there. (vinodkv) |
| |
| YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager |
| to separate out various validation checks depending on whether they rely on |
| RM configuration or not. (Zhijie Shen via vinodkv) |
| |
| YARN-618. Modified RM_INVALID_IDENTIFIER to be -1 instead of zero. (Jian He |
| via vinodkv) |
| |
| YARN-625. Move the utility method unwrapAndThrowException from |
| YarnRemoteExceptionPBImpl to RPCUtil. (Siddharth Seth via vinodkv) |
| |
| YARN-645. Moved RMDelegationTokenSecretManager from yarn-server-common to |
| yarn-server-resourcemanager where it really belongs. (Jian He via vinodkv) |
| |
| YARN-651. Changed PBClientImpls of ContainerManager and RMAdmin to throw |
| IOExceptions also. (Xuan Gong via vinodkv) |
| |
| YARN-582. Changed ResourceManager to recover Application token and client |
| tokens for app attempt so that RM can be restarted while preserving current |
| applications. (Jian He via vinodkv) |
| |
| YARN-568. Add support for work preserving preemption to the FairScheduler. |
| (Carlo Curino and Sandy Ryza via cdouglas) |
| |
| YARN-598. Add virtual cores to queue metrics. (sandyr via tucu) |
| |
| YARN-634. Modified YarnRemoteException to be not backed by PB and introduced |
| a separate SerializedException record. (Siddharth Seth via vinodkv) |
| |
| YARN-663. Changed ResourceTracker API and LocalizationProtocol API to throw |
| YarnRemoteException and IOException. (Xuan Gong via vinodkv) |
| |
| YARN-590. Added an optional mesage to be returned by ResourceMaanger when RM |
| asks an RM to shutdown/resync etc so that NMs can log this message locally |
| for better debuggability. (Mayank Bansal via vinodkv) |
| |
| YARN-617. Made ContainerTokens to be used for validation at NodeManager |
| also in unsecure mode to prevent AMs from faking resource requirements in |
| unsecure mode. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-708. Moved RecordFactory classes to hadoop-yarn-api, and put some |
| miscellaneous fixes to the interfaces. (Siddharth Seth via vinodkv) |
| |
| YARN-711. Copied BuilderUtil methods in individual API records as |
| BuilderUtils is going to be dismantled. (Jian He via vinodkv) |
| |
| YARN-714. Added NMTokens to be sent to AMs as part of heart-beat response. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-638. Modified ResourceManager to restore RMDelegationTokens after |
| restarting. (Jian He via vinodkv) |
| |
| YARN-660. Improve AMRMClient with matching requests (bikas) |
| |
| YARN-717. Put object creation factories for Token in the class itself and |
| remove useless derivations for specific tokens. (Jian He via vinodkv) |
| |
| YARN-756. Move Preemption* records to yarn.api where they really belong. |
| (Jian He via vinodkv) |
| |
| YARN-750. Allow for black-listing resources in YARN API and Impl in CS |
| (acmurthy via bikas) |
| |
| YARN-877. Support resource blacklisting for FifoScheduler. |
| (Junping Du via llu) |
| |
| YARN-686. Flatten NodeReport. (sandyr via tucu) |
| |
| YARN-737. Throw some specific exceptions directly instead of wrapping them |
| in YarnException. (Jian He via sseth) |
| |
| YARN-731. RPCUtil.unwrapAndThrowException should unwrap remote |
| RuntimeExceptions. (Zhijie Shen via sseth) |
| |
| YARN-600. Hook up cgroups CPU settings to the number of virtual cores |
| allocated. (sandyr via tucu) |
| |
| YARN-648. FS: Add documentation for pluggable policy. (kkambatl via tucu) |
| |
| YARN-773. Moved YarnRuntimeException from package api.yarn to |
| api.yarn.exceptions. (Jian He via vinodkv) |
| |
| YARN-692. Creating NMToken master key on RM and sharing it with NM as a part |
| of RM-NM heartbeat. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-782. vcores-pcores ratio functions differently from vmem-pmem ratio in |
| misleading way. (sandyr via tucu) |
| |
| YARN-803. factor out scheduler config validation from the ResourceManager |
| to each scheduler implementation. (tucu) |
| |
| YARN-789. Enable zero capabilities resource requests in fair scheduler. |
| (tucu) |
| |
| YARN-639. Modified Distributed Shell application to start using the new |
| NMClient library. (Zhijie Shen via vinodkv) |
| |
| YARN-693. Modified RM to send NMTokens on allocate call so that AMs can then |
| use them for authentication with NMs. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-752. In AMRMClient, automatically add corresponding rack requests for |
| requested nodes. (sandyr via tucu) |
| |
| YARN-825. Fixed javadoc and annotations for yarn-common module. (vinodkv) |
| |
| YARN-833. Moved Graph and VisualizeStateMachine into yarn.state package. |
| (Zhijie Shen via vinodkv) |
| |
| YARN-805. Fix javadoc and annotations on classes in the yarn-api |
| package. (Jian He via sseth) |
| |
| YARN-846. Move pb Impl classes from yarn-api to yarn-common. (Jian He via |
| vinodkv) |
| |
| YARN-827. Need to make Resource arithmetic methods accessible (Jian He via |
| bikas) |
| |
| YARN-866. Add test for class ResourceWeights. (ywskycn via tucu) |
| |
| YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) |
| |
| YARN-883. Expose Fair Scheduler-specific queue metrics. (sandyr via tucu) |
| |
| YARN-569. Add support for requesting and enforcing preemption requests via |
| a capacity monitor. (Carlo Curino, cdouglas) |
| |
| YARN-521. Augment AM - RM client module to be able to request containers |
| only at specific locations (Sandy Ryza via bikas) |
| |
| YARN-513. Create common proxy client for communicating with RM. (Xuan Gong |
| & Jian He via bikas) |
| |
| YARN-927. Change ContainerRequest to not have more than 1 container count |
| and remove StoreContainerRequest (bikas) |
| |
| YARN-922. Change FileSystemRMStateStore to use directories (Jian He via |
| bikas) |
| |
| YARN-865. RM webservices can't query based on application Types. (Xuan Gong |
| via hitesh) |
| |
| YARN-912. Move client facing exceptions to yarn-api module. (Mayank Bansal |
| via vinodkv) |
| |
| YARN-84. Use Builder to build RPC server. (Brandon Li via szetszwo) |
| |
| YARN-1046. Disable mem monitoring by default in MiniYARNCluster. (Karthik |
| Kambatla via Sandy Ryza) |
| |
| YARN-1045. Improve toString implementation for PBImpls. (Jian He via sseth) |
| |
| OPTIMIZATIONS |
| |
| YARN-512. Log aggregation root directory check is more expensive than it |
| needs to be. (Maysam Yabandeh via jlowe) |
| |
| YARN-719. Move RMIdentifier from Container to ContainerTokenIdentifier. |
| (Vinod Kumar Vavilapalli via sseth) |
| |
| BUG FIXES |
| |
| YARN-383. AMRMClientImpl should handle null rmClient in stop() |
| (Hitesh Shah via sseth) |
| |
| YARN-385. Add missing fields - location and #containers to |
| ResourceRequestPBImpl's toString(). (Sandy Ryza via sseth) |
| |
| YARN-377. Use the new StringUtils methods added by HADOOP-9252 and fix |
| TestContainersMonitor. (Chris Nauroth via szetszwo) |
| |
| YARN-391. Formatting fixes for LCEResourceHandler classes. |
| (Steve Loughran via sseth) |
| |
| YARN-390. ApplicationCLI and NodeCLI hard-coded platform-specific line |
| separator causes test failures on Windows. (Chris Nauroth via suresh) |
| |
| YARN-406. Fix TestRackResolver to function in networks where "host1" |
| resolves to a valid host. (Hitesh Shah via sseth) |
| |
| YARN-376. Fixes a bug which would prevent the NM knowing about completed |
| containers and applications. (Jason Lowe via sseth) |
| |
| YARN-196. Nodemanager should be more robust in handling connection failure |
| to ResourceManager when a cluster is started (Xuan Gong via hitesh) |
| |
| YARN-485. TestProcfsProcessTree#testProcessTree() doesn't wait long enough |
| for the process to die. (kkambatl via tucu) |
| |
| YARN-71. Fix the NodeManager to clean up local-dirs on restart. |
| (Xuan Gong via sseth) |
| |
| YARN-378. Fix RM to make the AM max attempts/retries to be configurable |
| per application by clients. (Zhijie Shen via vinodkv) |
| |
| YARN-498. Unmanaged AM launcher does not set various constants in env for |
| an AM, also does not handle failed AMs properly. (Hitesh Shah via bikas) |
| |
| YARN-496. Fair scheduler configs are refreshed inconsistently in |
| reinitialize. (Sandy Ryza via tomwhite) |
| |
| YARN-474. Fix CapacityScheduler to trigger application-activation when |
| am-resource-percent configuration is refreshed. (Zhijie Shen via vinodkv) |
| |
| YARN-209. Fix CapacityScheduler to trigger application-activation when |
| the cluster capacity changes. (Zhijie Shen via vinodkv) |
| |
| YARN-24. Nodemanager fails to start if log aggregation enabled and |
| namenode unavailable. (sandyr via tucu) |
| |
| YARN-515. Node Manager not getting the master key. (Robert Joseph Evans |
| via jlowe) |
| |
| YARN-382. SchedulerUtils improve way normalizeRequest sets the resource |
| capabilities. (Zhijie Shen via bikas) |
| |
| YARN-467. Modify public distributed cache to localize files such that no |
| local directory hits unix file count limits and thus prevent job failures. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-101. Fix NodeManager heartbeat processing to not lose track of completed |
| containers in case of dropped heartbeats. (Xuan Gong via vinodkv) |
| |
| YARN-538. RM address DNS lookup can cause unnecessary slowness on every JHS |
| page load. (sandyr via tucu) |
| |
| YARN-532. Change RMAdmin and Localization client protocol PB implementations |
| to implement closeable so that they can be stopped when needed via |
| RPC.stopProxy(). (Siddharth Seth via vinodkv) |
| |
| YARN-99. Modify private distributed cache to localize files such that no |
| local directory hits unix file count limits and thus prevent job failures. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-112. Fixed a race condition during localization that fails containers. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-534. Change RM restart recovery to also account for AM max-attempts |
| configuration after the restart. (Jian He via vinodkv) |
| |
| YARN-539. Addressed memory leak of LocalResource objects NM when a resource |
| localization fails. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-319. Submitting a job to a fair scheduler queue for which the user |
| does not have permission causes the client to wait forever. |
| (shenhong via tomwhite) |
| |
| YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather |
| than its host:port during scheduling which caused incorrect locality for |
| containers. (Roger Hoover via acmurthy) |
| |
| YARN-500. Fixed YARN webapps to not roll-over ports when explicitly asked |
| to use non-ephemeral ports. (Kenji Kikushima via vinodkv) |
| |
| YARN-518. Fair Scheduler's document link could be added to the hadoop 2.x |
| main doc page. (sandyr via tucu) |
| |
| YARN-476. ProcfsBasedProcessTree info message confuses users. |
| (sandyr via tucu) |
| |
| YARN-585. Fix failure in TestFairScheduler#testNotAllowSubmitApplication |
| caused by YARN-514. (Zhijie Shen via vinodkv) |
| |
| YARN-547. Fixed race conditions in public and private resource localization |
| which used to cause duplicate downloads. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-594. Update test and add comments in YARN-534 (Jian He via bikas) |
| |
| YARN-549. YarnClient.submitApplication should wait for application to be |
| accepted by the RM (Zhijie Shen via bikas) |
| |
| YARN-605. Fix failing unit test in TestNMWebServices when versionInfo has |
| parantheses like when running on a git checkout. (Hitesh Shah via vinodkv) |
| |
| YARN-289. Fair scheduler allows reservations that won't fit on node. |
| (Sandy Ryza via tomwhite) |
| |
| YARN-576. Modified ResourceManager to reject NodeManagers that don't satisy |
| minimum resource requirements. (Kenji Kikushima via vinodkv) |
| |
| YARN-646. Fix two typos in Fair Scheduler user guide. (Dapeng Sun via atm) |
| |
| YARN-507. Add interface visibility and stability annotations to FS |
| interfaces/classes. (kkambatl via tucu) |
| |
| YARN-637. FS: maxAssign is not honored. (kkambatl via tucu) |
| |
| YARN-655. Fair scheduler metrics should subtract allocated memory from |
| available memory. (sandyr via tucu) |
| |
| YARN-628. Fix the way YarnRemoteException is being unrolled to extract out |
| the underlying exception. (Siddharth Seth via vinodkv) |
| |
| YARN-695. Remove masterContainer and status unused fields from |
| ApplicationReportProto and fix bugs in ApplicationReportPBImpl. (Zhijie Shen |
| via vinodkv) |
| |
| YARN-706. Fixed race conditions in TestFSDownload. (Zhijie Shen via vinodkv). |
| |
| YARN-715. Fixed unit test failures - TestDistributedShell and |
| TestUnmanagedAMLauncher. (Vinod Kumar Vavilapalli via sseth) |
| |
| YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs. |
| (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-733. Fixed TestNMClient from failing occasionally. (Zhijie Shen via |
| vinodkv) |
| |
| YARN-730. Fix NMClientAsync to remove completed containers. (Zhijie Shen |
| via acmurthy) |
| |
| YARN-726. Fix queue & finish time fields in web-ui for ResourceManager. |
| (Mayank Bansal via acmurthy) |
| |
| YARN-757. Changed TestRMRestart to use the default scheduler to avoid test |
| failures. (Bikas Saha via vinodkv) |
| |
| YARN-742. Log aggregation causes a lot of redundant setPermission calls. |
| (jlowe via kihwal) |
| |
| YARN-764. blank Used Resources on Capacity Scheduler page (Nemon Lou via |
| tgraves) |
| |
| YARN-761. TestNMClientAsync fails sometimes (Zhijie Shen via bikas) |
| |
| YARN-760. NodeManager throws AvroRuntimeException on failed start. |
| (Niranjan Singh via jlowe) |
| |
| YARN-767. Initialize application metrics at RM bootup. (Jian He via |
| acmurthy) |
| |
| YARN-700. TestInfoBlock fails on Windows because of line ending missmatch. |
| (Ivan Mitic via cnauroth) |
| |
| YARN-117. Migrated rest of YARN to the new service model. (Steve Louhran via |
| vinodkv) |
| |
| YARN-812. Set default logger for application summary logger to |
| hadoop.root.logger. (sseth via acmurthy) |
| |
| YARN-848. Nodemanager does not register with RM using the fully qualified |
| hostname. (Hitesh Shah via sseth) |
| |
| YARN-854. Fixing YARN bugs that are failing applications in secure |
| environment. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-861. TestContainerManager is failing. (Vinod Kumar Vavilapalli via |
| hitesh) |
| |
| YARN-874. Making common RPC to switch to not switch to simple when other |
| mechanisms are enabled and thus fix YARN/MR test failures after HADOOP-9421. |
| (Daryn Sharp and Vinod Kumar Vavilapalli via vinodkv) |
| |
| YARN-845. RM crash with NPE on NODE_UPDATE (Mayank Bansal via bikas) |
| |
| YARN-369. Handle ( or throw a proper error when receiving) status updates |
| from application masters that have not registered (Mayank Bansal & |
| Abhishek Kapoor via bikas) |
| |
| YARN-541. getAllocatedContainers() is not returning all the allocated |
| containers (bikas) |
| |
| YARN-763. AMRMClientAsync should stop heartbeating after receiving |
| shutdown from RM (Xuan Gong via bikas) |
| |
| YARN-654. AMRMClient: Perform sanity checks for parameters of public |
| methods (Xuan Gong via bikas)" |
| |
| YARN-919. Document setting default heap sizes in yarn-env.sh (Mayank |
| Bansal via hitesh) |
| |
| YARN-795. Fair scheduler queue metrics should subtract allocated vCores from |
| available vCores. (ywskycn via tucu) |
| |
| YARN-799. Fix CgroupsLCEResourcesHandler to use /tasks instead of |
| /cgroup.procs. (Chris Riccomini via acmurthy) |
| |
| YARN-333. Schedulers cannot control the queue-name of an |
| application. (sandyr via tucu) |
| |
| YARN-368. Fixed a typo in error message in Auxiliary services. (Albert Chu |
| via vinodkv) |
| |
| YARN-295. Fixed a race condition in ResourceManager RMAppAttempt state |
| machine. (Mayank Bansal via vinodkv) |
| |
| YARN-523. Modified a test-case to validate container diagnostics on |
| localization failures. (Jian He via vinodkv) |
| |
| YARN-661. Fixed NM to cleanup users' local directories correctly when |
| starting up. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-820. Fixed an invalid state transition in NodeManager caused by failing |
| resource localization. (Mayank Bansal via vinodkv) |
| |
| YARN-62. Modified NodeManagers to avoid AMs from abusing container tokens for |
| repetitive container launches. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-814. Improving diagnostics when containers fail during launch due to |
| various reasons like invalid env etc. (Jian He via vinodkv) |
| |
| YARN-897. Ensure child queues are ordered correctly to account for |
| completed containers. (Djellel Eddine Difallah via acmurthy) |
| |
| YARN-853. Fixed CapacityScheduler's maximum-am-resource-percent to properly |
| work beyond refreshing queues. (Devaraj K via vinodkv) |
| |
| YARN-873. YARNClient.getApplicationReport(unknownAppId) returns a null |
| report (Xuan Gong via bikas) |
| |
| YARN-875. Application can hang if AMRMClientAsync callback thread has |
| exception (Xuan Gong via bikas) |
| |
| YARN-461. Fair scheduler should not accept apps with empty string queue name. |
| (ywskycn via tucu) |
| |
| YARN-968. RM admin commands don't work. (vinodkv via kihwal) |
| |
| YARN-688. Fixed NodeManager to properly cleanup containers when it is shut |
| down. (Jian He via vinodkv) |
| |
| YARN-960. Fixed ResourceManager to propagate client-submitted credentials |
| irrespective of security. (Daryn Sharp via vinodkv) |
| |
| YARN-937. Fix unmanaged AM in non-secure/secure setup post YARN-701. (tucu) |
| |
| YARN-932. TestResourceLocalizationService.testLocalizationInit can fail on |
| JDK7. (Karthik Kambatla via Sandy Ryza) |
| |
| YARN-961. Changed ContainerManager to enforce Token auth irrespective of |
| security. (Omkar Vinit Joshi via vinodkv) |
| |
| YARN-945. Removed setting of AMRMToken's service from ResourceManager |
| and changed client libraries do it all the time and correctly. (vinodkv) |
| |
| YARN-656. In scheduler UI, including reserved memory in Memory Total can |
| make it exceed cluster capacity. (Sandy Ryza) |
| |
| BREAKDOWN OF HADOOP-8562/YARN-191 SUBTASKS AND RELATED JIRAS |
| |
| YARN-158. Yarn creating package-info.java must not depend on sh. |
| (Chris Nauroth via suresh) |
| |
| YARN-176. Some YARN tests fail to find winutils. (Chris Nauroth via suresh) |
| |
| YARN-207. YARN distribution build fails on Windows. (Chris Nauroth via |
| suresh) |
| |
| YARN-199. Yarn cmd line scripts for windows. (Ivan Mitic via suresh) |
| |
| YARN-213. YARN build script would be more readable using abspath. |
| (Chris Nauroth via suresh) |
| |
| YARN-233. Added support for running containers in MS Windows to YARN. (Chris |
| Nauroth via acmurthy) |
| |
| YARN-234. Added support for process tree and resource calculator in MS Windows |
| to YARN. (Chris Nauroth via acmurthy) |
| |
| YARN-259. Fix LocalDirsHandlerService to use Path rather than URIs. (Xuan |
| Gong via acmurthy) |
| |
| YARN-316. YARN container launch may exceed maximum Windows command line |
| length due to long classpath. (Chris Nauroth via suresh) |
| |
| YARN-359. Fixing commands for container signalling in Windows. (Chris Nauroth |
| via vinodkv) |
| |
| YARN-506. Move to common utils FileUtil#setReadable/Writable/Executable and |
| FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) |
| |
| YARN-488. TestContainerManagerSecurity fails on Windows. (Chris Nauroth |
| via hitesh) |
| |
| YARN-490. TestDistributedShell fails on Windows. (Chris Nauroth via hitesh) |
| |
| YARN-491. TestContainerLogsPage fails on Windows. (Chris Nauroth via hitesh) |
| |
| YARN-487. Modify path manipulation in LocalDirsHandlerService to let |
| TestDiskFailures pass on Windows. (Chris Nauroth via vinodkv) |
| |
| YARN-593. container launch on Windows does not correctly populate |
| classpath with new process's environment variables and localized resources |
| (Chris Nauroth via bikas) |
| |
| YARN-493. Fixed some shell related flaws in YARN on Windows. (Chris Nauroth |
| via vinodkv) |
| |
| YARN-839. TestContainerLaunch.testContainerEnvVariables fails on Windows. |
| (Chuan Liu via cnauroth) |
| |
| YARN-597. TestFSDownload fails on Windows due to dependencies on |
| tar/gzip/jar tools. (Ivan Mitic via acmurthy) |
| |
| YARN-852. TestAggregatedLogFormat.testContainerLogsFileAccess fails on |
| Windows. (Chuan Liu via cnauroth) |
| |
| YARN-894. NodeHealthScriptRunner timeout checking is inaccurate on Windows. |
| (Chuan Liu via cnauroth) |
| |
| YARN-909. Disable TestLinuxContainerExecutorWithMocks on Windows. (Chuan Liu |
| via cnauroth) |
| |
| YARN-1043. Push all metrics consistently. (Jian He via acmurthy) |
| |
| YARN-1056. Remove dual use of string 'resourcemanager' in |
| yarn.resourcemanager.connect.{max.wait.secs|retry_interval.secs} |
| (Karthik Kambatla via acmurthy) |
| |
| Release 2.0.5-alpha - 06/06/2013 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| Release 2.0.4-alpha - 2013-04-25 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-429. capacity-scheduler config missing from yarn-test artifact. |
| (sseth via hitesh) |
| |
| YARN-470. Support a way to disable resource monitoring on the NodeManager. |
| (Siddharth Seth via hitesh) |
| |
| Release 2.0.3-alpha - 2013-02-06 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| YARN-145. Add a Web UI to the fair share scheduler. (Sandy Ryza via tomwhite) |
| |
| YARN-3. Add support for CPU isolation/monitoring of containers. |
| (adferguson via tucu) |
| |
| YARN-230. RM Restart phase 1 - includes support for saving/restarting all |
| applications on an RM bounce. (Bikas Saha via acmurthy) |
| |
| YARN-103. Add a yarn AM-RM client module. (Bikas Saha via sseth) |
| |
| YARN-286. Add a YARN ApplicationClassLoader. (tomwhite) |
| |
| YARN-2. Enhanced CapacityScheduler to account for CPU alongwith memory for |
| multi-dimensional resource scheduling. (acmurthy) |
| |
| YARN-328. Use token request messages defined in hadoop common. (suresh) |
| |
| YARN-231. RM Restart - Add FS-based persistent store implementation for |
| RMStateStore (Bikas Saha via hitesh) |
| |
| IMPROVEMENTS |
| |
| YARN-223. Update process tree instead of getting new process trees. |
| (Radim Kolar via llu) |
| |
| YARN-57. Allow process-tree based resource calculation et al. to be |
| pluggable to support it on multiple platforms. (Radim Kolar via acmurthy) |
| |
| YARN-78. Changed UnManagedAM application to use YarnClient. (Bikas Saha via |
| vinodkv) |
| |
| YARN-93. Fixed RM to propagate diagnostics from applications that have |
| finished but failed (Jason Lowe via vinodkv). |
| |
| YARN-28. Fixed TestCompositeService to not depend on test-order and thus |
| made it pass on JDK7 (Thomas Graves via vinodkv). |
| |
| YARN-82. Change the default local and log dirs to be based on |
| hadoop.tmp.dir and yarn.log.dir. (Hemanth Yamijala via sseth) |
| |
| YARN-53. Added the missing getGroups API to ResourceManager. (Bo Wang via |
| vinodkv) |
| |
| YARN-116. Add the ability to change the RM include/exclude file without |
| a restart. (xieguiming and Harsh J via sseth) |
| |
| YARN-23. FairScheduler: FSQueueSchedulable#updateDemand() - potential |
| redundant aggregation. (kkambatl via tucu) |
| |
| YARN-127. Move RMAdmin tool to its correct location - the client module. |
| (vinodkv) |
| |
| YARN-40. Provided support for missing YARN commands (Devaraj K and Vinod |
| Kumar Vavilapalli via vinodkv) |
| |
| YARN-33. Change LocalDirsHandlerService to validate the configured local and |
| log dirs. (Mayank Bansal via sseth) |
| |
| YARN-94. Modify DistributedShell to point to main-class by default, clean up |
| the help message, and hard-code the AM class. (Hitesh Shah via vinodkv) |
| |
| YARN-146. Add unit tests for computing fair share in the fair scheduler. |
| (Sandy Ryza via tomwhite) |
| |
| HADOOP-8911. CRLF characters in source and text files. |
| (Raja Aluri via suresh) |
| |
| YARN-136. Make ClientToAMTokenSecretManager part of RMContext (Vinod Kumar |
| Vavilapalli via sseth) |
| |
| YARN-183. Clean up fair scheduler code. (Sandy Ryza via tomwhite) |
| |
| YARN-129. Simplify classpath construction for mini YARN tests. (tomwhite) |
| |
| YARN-254. Update fair scheduler web UI for hierarchical queues. |
| (sandyr via tucu) |
| |
| YARN-315. Using the common security token protobuf definition from hadoop |
| common. (Suresh Srinivas via vinodkv) |
| |
| YARN-170. Change NodeManager stop to be reentrant. (Sandy Ryza via vinodkv) |
| |
| YARN-331. Fill in missing fair scheduler documentation. (sandyr via tucu) |
| |
| YARN-277. Use AMRMClient in DistributedShell to exemplify the approach. |
| (Bikas Saha via hitesh) |
| |
| YARN-360. Allow apps to concurrently register tokens for renewal. |
| (Daryn Sharp via sseth) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-131. Fix incorrect ACL properties in capacity scheduler documentation. |
| (Ahmed Radwan via sseth) |
| |
| YARN-102. Move the apache header to the top of the file in MemStore.java. |
| (Devaraj K via sseth) |
| |
| YARN-134. ClientToAMSecretManager creates keys without checking for |
| validity of the appID. (Vinod Kumar Vavilapalli via sseth) |
| |
| YARN-30. Fixed tests verifying web-services to work on JDK7. (Thomas Graves |
| via vinodkv) |
| |
| YARN-150. Fixes AppRejectedTransition does not unregister a rejected |
| app-attempt from the ApplicationMasterService (Bikas Saha via sseth) |
| |
| YARN-140. Add capacity-scheduler-default.xml to provide a default set of |
| configurations for the capacity scheduler. (ahmed via tucu) |
| |
| YARN-179. Fix some unit test failures. (Vinod Kumar Vavilapalli via sseth) |
| |
| YARN-181. Fixed eclipse settings broken by capacity-scheduler.xml move via |
| YARN-140. (Siddharth Seth via vinodkv) |
| |
| YARN-169. Update log4j.appender.EventCounter to use |
| org.apache.hadoop.log.metrics.EventCounter (Anthony Rojas via tomwhite) |
| |
| YARN-184. Remove unnecessary locking in fair scheduler, and address |
| findbugs excludes. (sandyr via tucu) |
| |
| YARN-224. Fair scheduler logs too many nodeUpdate INFO messages. |
| (Sandy Ryza via tomwhite) |
| |
| YARN-222. Fair scheduler should create queue for each user by default. |
| (Sandy Ryza via tomwhite) |
| |
| MAPREDUCE-4778. Fair scheduler event log is only written if directory |
| exists on HDFS. (Sandy Ryza via tomwhite) |
| |
| YARN-229. Remove old unused RM recovery code. (Bikas Saha via acmurthy) |
| |
| YARN-187. Add hierarchical queues to the fair scheduler. |
| (Sandy Ryza via tomwhite) |
| |
| YARN-72. NM should handle cleaning up containers when it shuts down. |
| (Sandy Ryza via tomwhite) |
| |
| YARN-267. Fix fair scheduler web UI. (Sandy Ryza via tomwhite) |
| |
| YARN-264. y.s.rm.DelegationTokenRenewer attempts to renew token even |
| after removing an app. (kkambatl via tucu) |
| |
| YARN-271. Fair scheduler hits IllegalStateException trying to reserve |
| different apps on same node. (Sandy Ryza via tomwhite) |
| |
| YARN-272. Fair scheduler log messages try to print objects without |
| overridden toString methods. (sandyr via tucu) |
| |
| YARN-278. Fair scheduler maxRunningApps config causes no apps to make |
| progress. (sandyr via tucu) |
| |
| YARN-282. Fair scheduler web UI double counts Apps Submitted. |
| (sandyr via tucu) |
| |
| YARN-283. Fair scheduler fails to get queue info without root prefix. |
| (sandyr via tucu) |
| |
| YARN-192. Node update causes NPE in the fair scheduler. |
| (Sandy Ryza via tomwhite) |
| |
| YARN-288. Fair scheduler queue doesn't accept any jobs when ACLs are |
| configured. (Sandy Ryza via tomwhite) |
| |
| YARN-300. After YARN-271, fair scheduler can infinite loop and not |
| schedule any application. (Sandy Ryza via tomwhite) |
| |
| YARN-301. Fair scheduler throws ConcurrentModificationException when |
| iterating over app's priorities. (Sandy Ryza via tomwhite) |
| |
| YARN-217. Fix RMAdmin protocol description to make it work in secure mode |
| also. (Devaraj K via vinodkv) |
| |
| YARN-253. Fixed container-launch to not fail when there are no local |
| resources to localize. (Tom White via vinodkv) |
| |
| YARN-330. Fix flakey test: TestNodeManagerShutdown#testKillContainersOnShutdown. |
| (Sandy Ryza via hitesh) |
| |
| YARN-335. Fair scheduler doesn't check whether rack needs containers |
| before assigning to node. (Sandy Ryza via tomwhite) |
| |
| YARN-336. Fair scheduler FIFO scheduling within a queue only allows 1 |
| app at a time. (Sandy Ryza via tomwhite) |
| |
| YARN-135. Client tokens should be per app-attempt, and should be |
| unregistered on App-finish. (vinodkv via sseth) |
| |
| YARN-302. Fair scheduler assignmultiple should default to false. (sandyr via tucu) |
| |
| YARN-372. Move InlineDispatcher from hadoop-yarn-server-resourcemanager to |
| hadoop-yarn-common (sseth via hitesh) |
| |
| YARN-370. Fix SchedulerUtils to correctly round up the resource for |
| containers. (Zhijie Shen via acmurthy) |
| |
| YARN-355. Fixes a bug where RM app submission could jam under load. |
| (Daryn Sharp via sseth) |
| |
| Release 2.0.2-alpha - 2012-09-07 |
| |
| YARN-9. Rename YARN_HOME to HADOOP_YARN_HOME. (vinodkv via acmurthy) |
| |
| NEW FEATURES |
| |
| YARN-1. Promote YARN to be a sub-project of Apache Hadoop. (acmurthy) |
| |
| IMPROVEMENTS |
| |
| YARN-29. Add a yarn-client module. (Vinod Kumar Vavilapalli via sseth) |
| |
| YARN-10. Fix DistributedShell module to not have a dependency on |
| hadoop-mapreduce-client-core. (Hitesh Shah via vinodkv) |
| |
| YARN-80. Add support for delaying rack-local containers in |
| CapacityScheduler. (acmurthy) |
| |
| YARN-137. Change the default YARN scheduler to be the CapacityScheduler. |
| (sseth via acmurthy) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-13. Fix pom versions for YARN in branch-2 (todd) |
| |
| MAPREDUCE-2374. "Text File Busy" errors launching MR tasks. (Andy Isaacson |
| via atm) |
| |
| YARN-12. Fix findbugs warnings in FairScheduler. (Junping Du via acmurthy) |
| |
| YARN-22. Fix ContainerLogs to work if the log-dir is specified as a URI. |
| (Mayank Bansal via sseth) |
| |
| YARN-37. Change TestRMAppTransitions to use the DrainDispatcher. |
| (Mayank Bansal via sseth) |
| |
| YARN-79. Implement close on all clients to YARN so that RPC clients don't |
| throw exceptions on shut-down. (Vinod Kumar Vavilapalli) |
| |
| YARN-42. Modify NM's non-aggregating logs' handler to stop properly so that |
| NMs don't get NPEs on startup errors. (Devaraj K via vinodkv) |
| |
| YARN-15. Updated default classpath for YARN applications to reflect split of |
| YARN into a sub-project. (Arun C Murthy via vinodkv) |
| |
| YARN-75. Modified ResourceManager's RMContainer to handle a valid RELEASE |
| event at RUNNING state. (Siddharth Seth via vinodkv) |
| |
| YARN-138. Ensure default values for minimum/maximum container sizes is |
| sane. (harsh & sseth via acmurthy) |
| |
| Release 0.23.9 - UNRELEASED |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| Release 0.23.8 - 2013-06-05 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| YARN-548. Add tests for YarnUncaughtExceptionHandler (Vadim Bondarev via |
| jeagles) |
| |
| BUG FIXES |
| |
| YARN-363. Add webapps/proxy directory without which YARN proxy-server fails |
| when started in stand-alone mode. (Kenji Kikushima via vinodkv) |
| |
| YARN-690. RM exits on token cancel/renew problems (daryn via bobby) |
| |
| Release 0.23.7 - 2013-04-18 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| YARN-133 Update web services docs for RM clusterMetrics (Ravi Prakash via |
| kihwal) |
| |
| YARN-249. Capacity Scheduler web page should show list of active users per |
| queue like it used to (in 1.x) (Ravi Prakash via tgraves) |
| |
| YARN-236. RM should point tracking URL to RM web page when app fails to |
| start (Jason Lowe via jeagles) |
| |
| YARN-269. Resource Manager not logging the health_check_script result when |
| taking it out (Jason Lowe via kihwal) |
| |
| YARN-227. Application expiration difficult to debug for end-users |
| (Jason Lowe via jeagles) |
| |
| YARN-443. allow OS scheduling priority of NM to be different than the |
| containers it launches (tgraves) |
| |
| YARN-468. coverage fix for org.apache.hadoop.yarn.server.webproxy.amfilter |
| (Aleksey Gorshkov via bobby) |
| |
| YARN-200. yarn log does not output all needed information, and is in a |
| binary format (Ravi Prakash via jlowe) |
| |
| YARN-525. make CS node-locality-delay refreshable (Thomas Graves via jlowe) |
| |
| OPTIMIZATIONS |
| |
| YARN-357. App submission should not be synchronized (daryn) |
| |
| BUG FIXES |
| |
| YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan |
| Gong via tgraves) |
| |
| YARN-364. AggregatedLogDeletionService can take too long to delete logs |
| (jlowe) |
| |
| YARN-362. Unexpected extra results when using webUI table search (Ravi |
| Prakash via jlowe) |
| |
| YARN-400. RM can return null application resource usage report leading to |
| NPE in client (Jason Lowe via tgraves) |
| |
| YARN-426. Failure to download a public resource prevents further downloads |
| (Jason Lowe via bobby) |
| |
| YARN-448. Remove unnecessary hflush from log aggregation (Kihwal Lee via |
| bobby) |
| |
| YARN-345. Many InvalidStateTransitonException errors for ApplicationImpl |
| in Node Manager (Robert Parker via jlowe) |
| |
| YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal |
| via bobby) |
| |
| YARN-460. CS user left in list of active users for the queue even when |
| application finished (tgraves) |
| |
| Release 0.23.6 - 2013-02-06 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| YARN-285. Added a temporary plugin interface for RM to be able to redirect |
| to JobHistory server for apps that it no longer tracks. (Derek Dagit via |
| vinodkv) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-188. Coverage fixing for CapacityScheduler (Aleksey Gorshkov via |
| bobby) |
| |
| YARN-214. RMContainerImpl does not handle event EXPIRE at state RUNNING |
| (jeagles via bobby) |
| |
| YARN-151. Browser thinks RM main page JS is taking too long |
| (Ravi Prakash via bobby) |
| |
| YARN-204. test coverage for org.apache.hadoop.tools (Aleksey Gorshkov via |
| bobby) |
| |
| YARN-251. Proxy URI generation fails for blank tracking URIs (Tom White |
| via jlowe) |
| |
| YARN-258. RM web page UI shows Invalid Date for start and finish times |
| (Ravi Prakash via jlowe) |
| |
| YARN-266. RM and JHS Web UIs are blank because AppsBlock is not escaping |
| string properly (Ravi Prakash via jlowe) |
| |
| YARN-280. RM does not reject app submission with invalid tokens |
| (Daryn Sharp via tgraves) |
| |
| YARN-225. Proxy Link in RM UI thows NPE in Secure mode |
| (Devaraj K via bobby) |
| |
| YARN-293. Node Manager leaks LocalizerRunner object for every Container |
| (Robert Joseph Evans via jlowe) |
| |
| YARN-50. Implement renewal / cancellation of Delegation Tokens |
| (Siddharth Seth via tgraves) |
| |
| YARN-320. RM should always be able to renew its own tokens. |
| (Daryn Sharp via sseth) |
| |
| YARN-325. RM CapacityScheduler can deadlock when getQueueInfo() is |
| called and a container is completing (Arun C Murthy via tgraves) |
| |
| YARN-334. Maven RAT plugin is not checking all source files (tgraves) |
| |
| YARN-354. WebAppProxyServer exits immediately after startup (Liang Xie via |
| jlowe) |
| |
| Release 0.23.5 - 2012-11-28 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| YARN-161. Fix multiple compiler warnings for unchecked operations in YARN |
| common. (Chris Nauroth via vinodkv) |
| |
| YARN-43. Fix TestResourceTrackerService to not depend on test order and thus |
| pass on JDK7. (Thomas Graves via vinodkv) |
| |
| YARN-32. Fix TestApplicationTokens to not depend on test order and thus pass |
| on JDK7. (vinodkv) |
| |
| YARN-186. Coverage fixing LinuxContainerExecutor (Aleksey Gorshkov via |
| bobby) |
| |
| YARN-216. Remove jquery theming support. (Robert Joseph Evans via jlowe) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-163. Retrieving container log via NM webapp can hang with multibyte |
| characters in log (jlowe via bobby) |
| |
| YARN-174. Modify NodeManager to pass the user's configuration even when |
| rebooting. (vinodkv) |
| |
| YARN-177. CapacityScheduler - adding a queue while the RM is running has |
| wacky results (acmurthy vai tgraves) |
| |
| YARN-178. Fix custom ProcessTree instance creation (Radim Kolar via bobby) |
| |
| YARN-180. Capacity scheduler - containers that get reserved create |
| container token to early (acmurthy and bobby) |
| |
| YARN-139. Interrupted Exception within AsyncDispatcher leads to user |
| confusion. (Vinod Kumar Vavilapalli via jlowe) |
| |
| YARN-165. RM should point tracking URL to RM web page for app when AM fails |
| (jlowe via bobby) |
| |
| YARN-159. RM web ui applications page should be sorted to display last app |
| first (tgraves via bobby) |
| |
| YARN-166. capacity scheduler doesn't allow capacity < 1.0 (tgraves via |
| bobby) |
| |
| YARN-189. Fixed a deadlock between RM's ApplicationMasterService and the |
| dispatcher. (Thomas Graves via vinodkv) |
| |
| YARN-202. Log Aggregation generates a storm of fsync() for namenode |
| (Kihwal Lee via bobby) |
| |
| YARN-201. Fix CapacityScheduler to be less conservative for starved |
| off-switch requests. (jlowe via acmurthy) |
| |
| YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. |
| (jlowe via jeagles) |
| |
| YARN-212. NM state machine ignores an APPLICATION_CONTAINER_FINISHED event |
| when it shouldn't (Nathan Roberts via jlowe) |
| |
| YARN-219. NM should aggregate logs when application finishes. (bobby) |
| |
| Release 0.23.4 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| Change package of YarnClient to org.apache.hadoop. (Bikas Saha via vinodkv) |
| |
| YARN-108. FSDownload can create cache directories with the wrong |
| permissions (Jason Lowe via bobby) |
| |
| YARN-57. Allow process-tree based resource calculation et al. to be |
| pluggable to support it on multiple platforms. (Radim Kolar via acmurthy) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| YARN-88. DefaultContainerExecutor can fail to set proper permissions. |
| (Jason Lowe via sseth) |
| |
| YARN-106. Nodemanager needs to set permissions of local directories (jlowe |
| via bobby) |
| |
| Release 0.23.3 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMAZATIONS |
| |
| BUG FIXES |
| |
| YARN-14. Symlinks to peer distributed cache files no longer work |
| (Jason Lowe via bobby) |
| |
| YARN-25. remove old aggregated logs (Robert Evans via tgraves) |
| |
| YARN-27. Failed refreshQueues due to misconfiguration prevents further |
| refreshing of queues (Arun Murthy via tgraves) |
| |
| MAPREDUCE-4323. NM leaks filesystems (Jason Lowe via jeagles) |
| |
| YARN-39. RM-NM secret-keys should be randomly generated and rolled every |
| so often. (vinodkv and sseth via sseth) |
| |
| YARN-31. Fix TestDelegationTokenRenewer to not depend on test order so as to |
| pass tests on jdk7. (Thomas Graves via vinodkv) |
| |
| YARN-63. RMNodeImpl is missing valid transitions from the UNHEALTHY state |
| (Jason Lowe via bobby) |
| |
| YARN-60. Fixed a bug in ResourceManager which causes all NMs to get NPEs and |
| thus causes all containers to be rejected. (vinodkv) |
| |
| YARN-66. aggregated logs permissions not set properly (tgraves via bobby) |
| |
| YARN-68. NodeManager will refuse to shutdown indefinitely due to container |
| log aggregation (daryn via bobby) |
| |
| YARN-87. NM ResourceLocalizationService does not set permissions of local |
| cache directories (Jason Lowe via tgraves) |
