layout: page displayTitle: Deploy Tez Client Plugin & Configurations title: Deploy Tez Client Plugin & Configurations description: Deploy Tez Client Plugin & Configurations license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Deploy Tez Client Plugin & Configurations

Deploy Tez Client Plugin

Append client jar to package which is set by ‘tez.lib.uris’.

In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by ‘tez.lib.uris’.

In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.

Update tez-site.xml to enable Uniffle.

Property Name	Default	Description
tez.am.launch.cmd-opts	-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster	enable remote shuffle service
tez.rss.coordinator.quorum	coordinatorIp1:19999,coordinatorIp2:19999	coordinator address

Note that the RssDAGAppMaster will automatically disable slow start (i.e., tez.shuffle-vertex-manager.min-src-fraction=1, tez.shuffle-vertex-manager.max-src-fraction=1).

Tez Specific Configurations

Property Name	Default	Description
tez.rss.avoid.recompute.succeeded.task	false	Whether to avoid recompute succeeded task when node is unhealthy or black-listed
tez.rss.client.max.buffer.size	3k	The max buffer size in map side. Control the size of each segment(WrappedBuffer) in the buffer.
tez.rss.client.batch.trigger.num	50	The max batch of buffers to send data in map side. Affect the number of blocks sent to the server in each batch, and may affect rss_worker_used_buffer_size
tez.rss.client.send.thread.num	5	The thread pool size for the client to send data to the server.
tez.shuffle.mode	remote	Use Remote Shuffle if the value is set to ‘remote’ or use default config value, or set ‘local’ to use local shuffle when needs to fall back.

Remote Spill (Experimental)

In cloud environment, VM may have very limited disk space and performance. This experimental feature allows reduce tasks to spill data to remote storage (e.g., hdfs)

Property Name	Default	Description
tez.rss.reduce.remote.spill.enable	false	Whether to use remote spill
tez.rss.remote.spill.storage.path		The remote spill path
tez.rss.reduce.remote.spill.replication	1	The replication number to spill data to Hadoop FS
tez.rss.reduce.remote.spill.retries	5	The retry number to spill data to Hadoop FS

Notice: this feature requires the MEMORY_LOCAL_HADOOP mode.