Helix Tutorial: Task Throttling

In this chapter, we'll learn how to control the parallel execution of tasks in the task framework.

Task Throttling Configuration

Helix can control the number of tasks that are executed in parallel according to multiple thresholds. Applications can set these thresholds in the following configuration items:

• JobConfig.ConcurrentTasksPerInstance The number of concurrent tasks in this job that are allowed to run on an instance.
• InstanceConfig.MAX_CONCURRENT_TASK The number of total concurrent tasks that are allowed to run on an instance.

Also see WorkflowConfig.ParallelJobs.

Job Priority for Task Throttling

Whenever there are too many tasks to be scheduled according to the threshold, Helix will prioritize the older jobs. The age of a job is calculated based on the job start time.