| <!DOCTYPE HTML> |
| <html lang="en"> |
| <head> |
| <!-- Generated by javadoc (17) on Tue May 28 17:21:12 UTC 2024 --> |
| <title>SparkContext (Spark 4.0.0-preview1 JavaDoc)</title> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> |
| <meta name="dc.created" content="2024-05-28"> |
| <meta name="description" content="declaration: package: org.apache.spark, class: SparkContext"> |
| <meta name="generator" content="javadoc/ClassWriterImpl"> |
| <link rel="stylesheet" type="text/css" href="../../../stylesheet.css" title="Style"> |
| <link rel="stylesheet" type="text/css" href="../../../script-dir/jquery-ui.min.css" title="Style"> |
| <link rel="stylesheet" type="text/css" href="../../../jquery-ui.overrides.css" title="Style"> |
| <script type="text/javascript" src="../../../script.js"></script> |
| <script type="text/javascript" src="../../../script-dir/jquery-3.6.1.min.js"></script> |
| <script type="text/javascript" src="../../../script-dir/jquery-ui.min.js"></script> |
| </head> |
| <body class="class-declaration-page"> |
| <script type="text/javascript">var evenRowColor = "even-row-color"; |
| var oddRowColor = "odd-row-color"; |
| var tableTab = "table-tab"; |
| var activeTableTab = "active-table-tab"; |
| var pathtoroot = "../../../"; |
| loadScripts(document, 'script');</script> |
| <noscript> |
| <div>JavaScript is disabled on your browser.</div> |
| </noscript> |
| <div class="flex-box"> |
| <header role="banner" class="flex-header"> |
| <nav role="navigation"> |
| <!-- ========= START OF TOP NAVBAR ======= --> |
| <div class="top-nav" id="navbar-top"> |
| <div class="skip-nav"><a href="#skip-navbar-top" title="Skip navigation links">Skip navigation links</a></div> |
| <ul id="navbar-top-firstrow" class="nav-list" title="Navigation"> |
| <li><a href="../../../index.html">Overview</a></li> |
| <li><a href="package-summary.html">Package</a></li> |
| <li class="nav-bar-cell1-rev">Class</li> |
| <li><a href="../../../deprecated-list.html">Deprecated</a></li> |
| <li><a href="../../../index-all.html">Index</a></li> |
| <li><a href="../../../help-doc.html#class">Help</a></li> |
| </ul> |
| </div> |
| <div class="sub-nav"> |
| <div> |
| <ul class="sub-nav-list"> |
| <li>Summary: </li> |
| <li><a href="#nested-class-summary">Nested</a> | </li> |
| <li>Field | </li> |
| <li><a href="#constructor-summary">Constr</a> | </li> |
| <li><a href="#method-summary">Method</a></li> |
| </ul> |
| <ul class="sub-nav-list"> |
| <li>Detail: </li> |
| <li>Field | </li> |
| <li><a href="#constructor-detail">Constr</a> | </li> |
| <li><a href="#method-detail">Method</a></li> |
| </ul> |
| </div> |
| <div class="nav-list-search"><label for="search-input">SEARCH:</label> |
| <input type="text" id="search-input" value="search" disabled="disabled"> |
| <input type="reset" id="reset-button" value="reset" disabled="disabled"> |
| </div> |
| </div> |
| <!-- ========= END OF TOP NAVBAR ========= --> |
| <span class="skip-nav" id="skip-navbar-top"></span></nav> |
| </header> |
| <div class="flex-content"> |
| <main role="main"> |
| <!-- ======== START OF CLASS DATA ======== --> |
| <div class="header"> |
| <div class="sub-title"><span class="package-label-in-type">Package</span> <a href="package-summary.html">org.apache.spark</a></div> |
| <h1 title="Class SparkContext" class="title">Class SparkContext</h1> |
| </div> |
| <div class="inheritance" title="Inheritance Tree"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a> |
| <div class="inheritance">org.apache.spark.SparkContext</div> |
| </div> |
| <section class="class-description" id="class-description"> |
| <dl class="notes"> |
| <dt>All Implemented Interfaces:</dt> |
| <dd><code>org.apache.spark.internal.Logging</code></dd> |
| </dl> |
| <hr> |
| <div class="type-signature"><span class="modifiers">public class </span><span class="element-name type-name-label">SparkContext</span> |
| <span class="extends-implements">extends <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a> |
| implements org.apache.spark.internal.Logging</span></div> |
| <div class="block">Main entry point for Spark functionality. A SparkContext represents the connection to a Spark |
| cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster. |
| <p></div> |
| <dl class="notes"> |
| <dt>Note:</dt> |
| <dd>Only one <code>SparkContext</code> should be active per JVM. You must <code>stop()</code> the |
| active <code>SparkContext</code> before creating a new one. |
| param: config a Spark Config object describing the application configuration. Any settings in |
| this config overrides the default configs as well as system properties.</dd> |
| </dl> |
| </section> |
| <section class="summary"> |
| <ul class="summary-list"> |
| <!-- ======== NESTED CLASS SUMMARY ======== --> |
| <li> |
| <section class="nested-class-summary" id="nested-class-summary"> |
| <h2>Nested Class Summary</h2> |
| <div class="inherited-list"> |
| <h2 id="nested-classes-inherited-from-class-org.apache.spark.internal.Logging">Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging</h2> |
| <code>org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter</code></div> |
| </section> |
| </li> |
| <!-- ======== CONSTRUCTOR SUMMARY ======== --> |
| <li> |
| <section class="constructor-summary" id="constructor-summary"> |
| <h2>Constructor Summary</h2> |
| <div class="caption"><span>Constructors</span></div> |
| <div class="summary-table two-column-summary"> |
| <div class="table-header col-first">Constructor</div> |
| <div class="table-header col-last">Description</div> |
| <div class="col-constructor-name even-row-color"><code><a href="#%3Cinit%3E()" class="member-name-link">SparkContext</a>()</code></div> |
| <div class="col-last even-row-color"> |
| <div class="block">Create a SparkContext that loads settings from system properties (for instance, when |
| launching with ./bin/spark-submit).</div> |
| </div> |
| <div class="col-constructor-name odd-row-color"><code><a href="#%3Cinit%3E(java.lang.String,java.lang.String,java.lang.String,scala.collection.immutable.Seq,scala.collection.Map)" class="member-name-link">SparkContext</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> master, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> appName, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> sparkHome, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>> jars, |
| scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>> environment)</code></div> |
| <div class="col-last odd-row-color"> |
| <div class="block">Alternative constructor that allows setting common Spark properties directly</div> |
| </div> |
| <div class="col-constructor-name even-row-color"><code><a href="#%3Cinit%3E(java.lang.String,java.lang.String,org.apache.spark.SparkConf)" class="member-name-link">SparkContext</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> master, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> appName, |
| <a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a> conf)</code></div> |
| <div class="col-last even-row-color"> |
| <div class="block">Alternative constructor that allows setting common Spark properties directly</div> |
| </div> |
| <div class="col-constructor-name odd-row-color"><code><a href="#%3Cinit%3E(org.apache.spark.SparkConf)" class="member-name-link">SparkContext</a><wbr>(<a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a> config)</code></div> |
| <div class="col-last odd-row-color"> </div> |
| </div> |
| </section> |
| </li> |
| <!-- ========== METHOD SUMMARY =========== --> |
| <li> |
| <section class="method-summary" id="method-summary"> |
| <h2>Method Summary</h2> |
| <div id="method-summary-table"> |
| <div class="table-tabs" role="tablist" aria-orientation="horizontal"><button id="method-summary-table-tab0" role="tab" aria-selected="true" aria-controls="method-summary-table.tabpanel" tabindex="0" onkeydown="switchTab(event)" onclick="show('method-summary-table', 'method-summary-table', 3)" class="active-table-tab">All Methods</button><button id="method-summary-table-tab1" role="tab" aria-selected="false" aria-controls="method-summary-table.tabpanel" tabindex="-1" onkeydown="switchTab(event)" onclick="show('method-summary-table', 'method-summary-table-tab1', 3)" class="table-tab">Static Methods</button><button id="method-summary-table-tab2" role="tab" aria-selected="false" aria-controls="method-summary-table.tabpanel" tabindex="-1" onkeydown="switchTab(event)" onclick="show('method-summary-table', 'method-summary-table-tab2', 3)" class="table-tab">Instance Methods</button><button id="method-summary-table-tab4" role="tab" aria-selected="false" aria-controls="method-summary-table.tabpanel" tabindex="-1" onkeydown="switchTab(event)" onclick="show('method-summary-table', 'method-summary-table-tab4', 3)" class="table-tab">Concrete Methods</button></div> |
| <div id="method-summary-table.tabpanel" role="tabpanel"> |
| <div class="summary-table three-column-summary" aria-labelledby="method-summary-table-tab0"> |
| <div class="table-header col-first">Modifier and Type</div> |
| <div class="table-header col-second">Method</div> |
| <div class="table-header col-last">Description</div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#addArchive(java.lang.String)" class="member-name-link">addArchive</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: Experimental :: |
| Add an archive to be downloaded and unpacked with this Spark job on every node.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#addFile(java.lang.String)" class="member-name-link">addFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Add a file to be downloaded with this Spark job on every node.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#addFile(java.lang.String,boolean)" class="member-name-link">addFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| boolean recursive)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Add a file to be downloaded with this Spark job on every node.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#addJar(java.lang.String)" class="member-name-link">addJar</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Adds a JAR dependency for all tasks to be executed on this <code>SparkContext</code> in the future.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#addJobTag(java.lang.String)" class="member-name-link">addJobTag</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> tag)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Add a tag to be assigned to all the jobs started by this thread.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#addSparkListener(org.apache.spark.scheduler.SparkListenerInterface)" class="member-name-link">addSparkListener</a><wbr>(<a href="scheduler/SparkListenerInterface.html" title="interface in org.apache.spark.scheduler">SparkListenerInterface</a> listener)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Register a listener to receive up-calls from events that happen during execution.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#applicationAttemptId()" class="member-name-link">applicationAttemptId</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#applicationId()" class="member-name-link">applicationId</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">A unique identifier for the Spark application.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#appName()" class="member-name-link">appName</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#archives()" class="member-name-link">archives</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="input/PortableDataStream.html" title="class in org.apache.spark.input">PortableDataStream</a>>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#binaryFiles(java.lang.String,int)" class="member-name-link">binaryFiles</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file |
| (useful for binary data)</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><byte[]></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#binaryRecords(java.lang.String,int,org.apache.hadoop.conf.Configuration)" class="member-name-link">binaryRecords</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int recordLength, |
| org.apache.hadoop.conf.Configuration conf)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Load data from a flat binary file, assuming the length of each record is constant.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="broadcast/Broadcast.html" title="class in org.apache.spark.broadcast">Broadcast</a><T></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#broadcast(T,scala.reflect.ClassTag)" class="member-name-link">broadcast</a><wbr>(T value, |
| scala.reflect.ClassTag<T> evidence$9)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Broadcast a read-only variable to the cluster, returning a |
| <a href="broadcast/Broadcast.html" title="class in org.apache.spark.broadcast"><code>Broadcast</code></a> object for reading it in distributed functions.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelAllJobs()" class="member-name-link">cancelAllJobs</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel all jobs that have been scheduled or are running.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelJob(int)" class="member-name-link">cancelJob</a><wbr>(int jobId)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel a given job if it's scheduled or running.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelJob(int,java.lang.String)" class="member-name-link">cancelJob</a><wbr>(int jobId, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> reason)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel a given job if it's scheduled or running.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelJobGroup(java.lang.String)" class="member-name-link">cancelJobGroup</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> groupId)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel active jobs for the specified group.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelJobGroupAndFutureJobs(java.lang.String)" class="member-name-link">cancelJobGroupAndFutureJobs</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> groupId)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel active jobs for the specified group, as well as the future jobs in this job group.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelJobsWithTag(java.lang.String)" class="member-name-link">cancelJobsWithTag</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> tag)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel active jobs that have the specified tag.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelStage(int)" class="member-name-link">cancelStage</a><wbr>(int stageId)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel a given stage and all jobs associated with it.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#cancelStage(int,java.lang.String)" class="member-name-link">cancelStage</a><wbr>(int stageId, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> reason)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Cancel a given stage and all jobs associated with it.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#clearCallSite()" class="member-name-link">clearCallSite</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Clear the thread-local property for overriding the call sites |
| of actions and RDDs.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#clearJobGroup()" class="member-name-link">clearJobGroup</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Clear the current thread's job group ID and its description.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#clearJobTags()" class="member-name-link">clearJobTags</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Clear the current thread's job tags.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="util/CollectionAccumulator.html" title="class in org.apache.spark.util">CollectionAccumulator</a><T></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#collectionAccumulator()" class="member-name-link">collectionAccumulator</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Create and register a <code>CollectionAccumulator</code>, which starts with empty list and accumulates |
| inputs by adding them into the list.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="util/CollectionAccumulator.html" title="class in org.apache.spark.util">CollectionAccumulator</a><T></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#collectionAccumulator(java.lang.String)" class="member-name-link">collectionAccumulator</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Create and register a <code>CollectionAccumulator</code>, which starts with empty list and accumulates |
| inputs by adding them into the list.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>int</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#defaultMinPartitions()" class="member-name-link">defaultMinPartitions</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Default min number of partitions for Hadoop RDDs when not given by user |
| Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>int</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#defaultParallelism()" class="member-name-link">defaultParallelism</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Default level of parallelism to use when not given by user (e.g.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#deployMode()" class="member-name-link">deployMode</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="util/DoubleAccumulator.html" title="class in org.apache.spark.util">DoubleAccumulator</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#doubleAccumulator()" class="member-name-link">doubleAccumulator</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Create and register a double accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="util/DoubleAccumulator.html" title="class in org.apache.spark.util">DoubleAccumulator</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#doubleAccumulator(java.lang.String)" class="member-name-link">doubleAccumulator</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Create and register a double accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#emptyRDD(scala.reflect.ClassTag)" class="member-name-link">emptyRDD</a><wbr>(scala.reflect.ClassTag<T> evidence$8)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD that has no partitions or elements.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#files()" class="member-name-link">files</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Seq<<a href="scheduler/Schedulable.html" title="interface in org.apache.spark.scheduler">Schedulable</a>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getAllPools()" class="member-name-link">getAllPools</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Return pools for fair scheduler</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getCheckpointDir()" class="member-name-link">getCheckpointDir</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getConf()" class="member-name-link">getConf</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Return a copy of this SparkContext's configuration.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr>scala.Tuple2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getExecutorMemoryStatus()" class="member-name-link">getExecutorMemoryStatus</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Return a map from the block manager to the max memory available for caching and the remaining |
| memory available for caching.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Set<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getJobTags()" class="member-name-link">getJobTags</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get the tags that are currently set to be assigned to all the jobs started by this thread.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getLocalProperty(java.lang.String)" class="member-name-link">getLocalProperty</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> key)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get a local property set in this thread, or null if it is missing.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code>static <a href="SparkContext.html" title="class in org.apache.spark">SparkContext</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code><a href="#getOrCreate()" class="member-name-link">getOrCreate</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"> |
| <div class="block">This function may be used to get or instantiate a SparkContext and register it as a |
| singleton object.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code>static <a href="SparkContext.html" title="class in org.apache.spark">SparkContext</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code><a href="#getOrCreate(org.apache.spark.SparkConf)" class="member-name-link">getOrCreate</a><wbr>(<a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a> config)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"> |
| <div class="block">This function may be used to get or instantiate a SparkContext and register it as a |
| singleton object.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><?>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getPersistentRDDs()" class="member-name-link">getPersistentRDDs</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.Option<<a href="scheduler/Schedulable.html" title="interface in org.apache.spark.scheduler">Schedulable</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getPoolForName(java.lang.String)" class="member-name-link">getPoolForName</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> pool)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Return the pool associated with the given name, if one exists</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="storage/RDDInfo.html" title="class in org.apache.spark.storage">RDDInfo</a>[]</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getRDDStorageInfo()" class="member-name-link">getRDDStorageInfo</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Return information about what RDDs are cached, if they are in mem or on disk, how much space |
| they take, etc.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.Enumeration.Value</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#getSchedulingMode()" class="member-name-link">getSchedulingMode</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Return current scheduling mode</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>org.apache.hadoop.conf.Configuration</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#hadoopConfiguration()" class="member-name-link">hadoopConfiguration</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">A default Hadoop Configuration for the Hadoop code (e.g.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapred.InputFormat<K,<wbr> |
| V>><br><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#hadoopFile(java.lang.String,int,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.reflect.ClassTag)" class="member-name-link">hadoopFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.reflect.ClassTag<F> fm)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, |
| values and the InputFormat so that users don't need to pass them directly.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#hadoopFile(java.lang.String,java.lang.Class,java.lang.Class,java.lang.Class,int)" class="member-name-link">hadoopFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><? extends org.apache.hadoop.mapred.InputFormat<K,<wbr>V>> inputFormatClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass, |
| int minPartitions)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD for a Hadoop file with an arbitrary InputFormat</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapred.InputFormat<K,<wbr> |
| V>><br><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#hadoopFile(java.lang.String,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.reflect.ClassTag)" class="member-name-link">hadoopFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.reflect.ClassTag<F> fm)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, |
| values and the InputFormat so that users don't need to pass them directly.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#hadoopRDD(org.apache.hadoop.mapred.JobConf,java.lang.Class,java.lang.Class,java.lang.Class,int)" class="member-name-link">hadoopRDD</a><wbr>(org.apache.hadoop.mapred.JobConf conf, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><? extends org.apache.hadoop.mapred.InputFormat<K,<wbr>V>> inputFormatClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass, |
| int minPartitions)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other |
| necessary info (e.g.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>boolean</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#isLocal()" class="member-name-link">isLocal</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>boolean</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#isStopped()" class="member-name-link">isStopped</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code>static scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code><a href="#jarOfClass(java.lang.Class)" class="member-name-link">jarOfClass</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><?> cls)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"> |
| <div class="block">Find the JAR from which a given class was loaded, to make it easy for users to pass |
| their JARs to SparkContext.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code>static scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code><a href="#jarOfObject(java.lang.Object)" class="member-name-link">jarOfObject</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a> obj)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"> |
| <div class="block">Find the JAR that contains the class of a particular object, to make it easy for users |
| to pass their JARs to SparkContext.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#jars()" class="member-name-link">jars</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>boolean</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#killExecutor(java.lang.String)" class="member-name-link">killExecutor</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> executorId)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Request that the cluster manager kill the specified executor.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>boolean</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#killExecutors(scala.collection.immutable.Seq)" class="member-name-link">killExecutors</a><wbr>(scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>> executorIds)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Request that the cluster manager kill the specified executors.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>boolean</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#killTaskAttempt(long,boolean,java.lang.String)" class="member-name-link">killTaskAttempt</a><wbr>(long taskId, |
| boolean interruptThread, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> reason)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Kill and reschedule the given task attempt.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#listArchives()" class="member-name-link">listArchives</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: Experimental :: |
| Returns a list of archive paths that are added to resources.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#listFiles()" class="member-name-link">listFiles</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Returns a list of file paths that are added to resources.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#listJars()" class="member-name-link">listJars</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Returns a list of jar files that are added to resources.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code>static org.apache.spark.internal.Logging.LogStringContext</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code><a href="#LogStringContext(scala.StringContext)" class="member-name-link">LogStringContext</a><wbr>(scala.StringContext sc)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"> </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="util/LongAccumulator.html" title="class in org.apache.spark.util">LongAccumulator</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#longAccumulator()" class="member-name-link">longAccumulator</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Create and register a long accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="util/LongAccumulator.html" title="class in org.apache.spark.util">LongAccumulator</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#longAccumulator(java.lang.String)" class="member-name-link">longAccumulator</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Create and register a long accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#makeRDD(scala.collection.immutable.Seq,scala.reflect.ClassTag)" class="member-name-link">makeRDD</a><wbr>(scala.collection.immutable.Seq<scala.Tuple2<T,<wbr>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>>>> seq, |
| scala.reflect.ClassTag<T> evidence$3)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Distribute a local Scala collection to form an RDD, with one or more |
| location preferences (hostnames of Spark nodes) for each object.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#makeRDD(scala.collection.immutable.Seq,int,scala.reflect.ClassTag)" class="member-name-link">makeRDD</a><wbr>(scala.collection.immutable.Seq<T> seq, |
| int numSlices, |
| scala.reflect.ClassTag<T> evidence$2)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Distribute a local Scala collection to form an RDD.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#master()" class="member-name-link">master</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapreduce.InputFormat<K,<wbr> |
| V>><br><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#newAPIHadoopFile(java.lang.String,java.lang.Class,java.lang.Class,java.lang.Class,org.apache.hadoop.conf.Configuration)" class="member-name-link">newAPIHadoopFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><F> fClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> kClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> vClass, |
| org.apache.hadoop.conf.Configuration conf)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD for a given Hadoop file with an arbitrary new API InputFormat |
| and extra configuration options to pass to the input format.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapreduce.InputFormat<K,<wbr> |
| V>><br><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#newAPIHadoopFile(java.lang.String,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.reflect.ClassTag)" class="member-name-link">newAPIHadoopFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.reflect.ClassTag<F> fm)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Smarter version of <code>newApiHadoopFile</code> that uses class tags to figure out the classes of keys, |
| values and the <code>org.apache.hadoop.mapreduce.InputFormat</code> (new MapReduce API) so that user |
| don't need to pass them directly.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapreduce.InputFormat<K,<wbr> |
| V>><br><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#newAPIHadoopRDD(org.apache.hadoop.conf.Configuration,java.lang.Class,java.lang.Class,java.lang.Class)" class="member-name-link">newAPIHadoopRDD</a><wbr>(org.apache.hadoop.conf.Configuration conf, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><F> fClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> kClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> vClass)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD for a given Hadoop file with an arbitrary new API InputFormat |
| and extra configuration options to pass to the input format.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#objectFile(java.lang.String,int,scala.reflect.ClassTag)" class="member-name-link">objectFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions, |
| scala.reflect.ClassTag<T> evidence$4)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and |
| BytesWritable values that contain a serialized partition.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code>static org.slf4j.Logger</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code><a href="#org$apache$spark$internal$Logging$$log_()" class="member-name-link">org$apache$spark$internal$Logging$$log_</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"> </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code>static void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"><code><a href="#org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger)" class="member-name-link">org$apache$spark$internal$Logging$$log__$eq</a><wbr>(org.slf4j.Logger x$1)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab1 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#parallelize(scala.collection.immutable.Seq,int,scala.reflect.ClassTag)" class="member-name-link">parallelize</a><wbr>(scala.collection.immutable.Seq<T> seq, |
| int numSlices, |
| scala.reflect.ClassTag<T> evidence$1)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Distribute a local Scala collection to form an RDD.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#range(long,long,long,int)" class="member-name-link">range</a><wbr>(long start, |
| long end, |
| long step, |
| int numSlices)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Creates a new RDD[Long] containing elements from <code>start</code> to <code>end</code>(exclusive), increased by |
| <code>step</code> every element.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#register(org.apache.spark.util.AccumulatorV2)" class="member-name-link">register</a><wbr>(<a href="util/AccumulatorV2.html" title="class in org.apache.spark.util">AccumulatorV2</a><?,<wbr>?> acc)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Register the given accumulator.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#register(org.apache.spark.util.AccumulatorV2,java.lang.String)" class="member-name-link">register</a><wbr>(<a href="util/AccumulatorV2.html" title="class in org.apache.spark.util">AccumulatorV2</a><?,<wbr>?> acc, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Register the given accumulator with given name.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#removeJobTag(java.lang.String)" class="member-name-link">removeJobTag</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> tag)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Remove a tag previously added to be assigned to all the jobs started by this thread.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#removeSparkListener(org.apache.spark.scheduler.SparkListenerInterface)" class="member-name-link">removeSparkListener</a><wbr>(<a href="scheduler/SparkListenerInterface.html" title="interface in org.apache.spark.scheduler">SparkListenerInterface</a> listener)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Deregister the listener from Spark's listener bus.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>boolean</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#requestExecutors(int)" class="member-name-link">requestExecutors</a><wbr>(int numAdditionalExecutors)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Request an additional number of executors from the cluster manager.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>boolean</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#requestTotalExecutors(int,int,scala.collection.immutable.Map)" class="member-name-link">requestTotalExecutors</a><wbr>(int numExecutors, |
| int localityAwareTasks, |
| scala.collection.immutable.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> hostToLocalTaskCount)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Update the cluster manager on our scheduling needs.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="resource/ResourceInformation.html" title="class in org.apache.spark.resource">ResourceInformation</a>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#resources()" class="member-name-link">resources</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U,<wbr> |
| R> <a href="partial/PartialResult.html" title="class in org.apache.spark.partial">PartialResult</a><R></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runApproximateJob(org.apache.spark.rdd.RDD,scala.Function2,org.apache.spark.partial.ApproximateEvaluator,long)" class="member-name-link">runApproximateJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| <a href="partial/ApproximateEvaluator.html" title="interface in org.apache.spark.partial">ApproximateEvaluator</a><U,<wbr>R> evaluator, |
| long timeout)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">:: DeveloperApi :: |
| Run a job that can return approximate results.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U> <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runJob(org.apache.spark.rdd.RDD,scala.Function1,scala.collection.immutable.Seq,scala.reflect.ClassTag)" class="member-name-link">runJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> func, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.reflect.ClassTag<U> evidence$13)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Run a function on a given set of partitions in an RDD and return the results as an array.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U> void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runJob(org.apache.spark.rdd.RDD,scala.Function1,scala.Function2,scala.reflect.ClassTag)" class="member-name-link">runJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> processPartition, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.reflect.ClassTag<U> evidence$17)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Run a job on all partitions in an RDD and pass the results to a handler function.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U> <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runJob(org.apache.spark.rdd.RDD,scala.Function1,scala.reflect.ClassTag)" class="member-name-link">runJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> func, |
| scala.reflect.ClassTag<U> evidence$15)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Run a job on all partitions in an RDD and return the results in an array.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U> void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.collection.immutable.Seq,scala.Function2,scala.reflect.ClassTag)" class="member-name-link">runJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.reflect.ClassTag<U> evidence$11)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Run a function on a given set of partitions in an RDD and pass the results to the given |
| handler function.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U> <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.collection.immutable.Seq,scala.reflect.ClassTag)" class="member-name-link">runJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.reflect.ClassTag<U> evidence$12)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Run a function on a given set of partitions in an RDD and return the results as an array.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U> void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.Function2,scala.reflect.ClassTag)" class="member-name-link">runJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> processPartition, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.reflect.ClassTag<U> evidence$16)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Run a job on all partitions in an RDD and pass the results to a handler function.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U> <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.reflect.ClassTag)" class="member-name-link">runJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| scala.reflect.ClassTag<U> evidence$14)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Run a job on all partitions in an RDD and return the results in an array.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#sequenceFile(java.lang.String,int,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.Function0,scala.Function0)" class="member-name-link">sequenceFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.Function0<org.apache.spark.WritableConverter<K>> kcf, |
| scala.Function0<org.apache.spark.WritableConverter<V>> vcf)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Version of sequenceFile() for types implicitly convertible to Writables through a |
| WritableConverter.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#sequenceFile(java.lang.String,java.lang.Class,java.lang.Class)" class="member-name-link">sequenceFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD for a Hadoop SequenceFile with given key and value types.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><K,<wbr> |
| V> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#sequenceFile(java.lang.String,java.lang.Class,java.lang.Class,int)" class="member-name-link">sequenceFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass, |
| int minPartitions)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Get an RDD for a Hadoop SequenceFile with given key and value types.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#setCallSite(java.lang.String)" class="member-name-link">setCallSite</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> shortCallSite)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Set the thread-local property for overriding the call sites |
| of actions and RDDs.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#setCheckpointDir(java.lang.String)" class="member-name-link">setCheckpointDir</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> directory)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Set the directory under which RDDs are going to be checkpointed.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#setInterruptOnCancel(boolean)" class="member-name-link">setInterruptOnCancel</a><wbr>(boolean interruptOnCancel)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Set the behavior of job cancellation from jobs started in this thread.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#setJobDescription(java.lang.String)" class="member-name-link">setJobDescription</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> value)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Set a human readable description of the current job.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#setJobGroup(java.lang.String,java.lang.String,boolean)" class="member-name-link">setJobGroup</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> groupId, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> description, |
| boolean interruptOnCancel)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Assigns a group ID to all the jobs started by this thread until the group ID is set to a |
| different value or cleared.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#setLocalProperty(java.lang.String,java.lang.String)" class="member-name-link">setLocalProperty</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> key, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> value)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Set a local property that affects jobs submitted from this thread, such as the Spark fair |
| scheduler pool.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#setLogLevel(java.lang.String)" class="member-name-link">setLogLevel</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> logLevel)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Control our logLevel.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#sparkUser()" class="member-name-link">sparkUser</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>long</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#startTime()" class="member-name-link">startTime</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="SparkStatusTracker.html" title="class in org.apache.spark">SparkStatusTracker</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#statusTracker()" class="member-name-link">statusTracker</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#stop()" class="member-name-link">stop</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Shut down the SparkContext.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>void</code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#stop(int)" class="member-name-link">stop</a><wbr>(int exitCode)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Shut down the SparkContext with exit code that will passed to scheduler backend.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T,<wbr> |
| U,<wbr> |
| R> <a href="SimpleFutureAction.html" title="class in org.apache.spark">SimpleFutureAction</a><R></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#submitJob(org.apache.spark.rdd.RDD,scala.Function1,scala.collection.immutable.Seq,scala.Function2,scala.Function0)" class="member-name-link">submitJob</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> processPartition, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.Function0<R> resultFunc)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Submit a job for execution and return a FutureJob holding the result.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#textFile(java.lang.String,int)" class="member-name-link">textFile</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Read a text file from HDFS, a local file system (available on all nodes), or any |
| Hadoop-supported file system URI, and return it as an RDD of Strings.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code>scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#uiWebUrl()" class="member-name-link">uiWebUrl</a>()</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#union(org.apache.spark.rdd.RDD,scala.collection.immutable.Seq,scala.reflect.ClassTag)" class="member-name-link">union</a><wbr>(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> first, |
| scala.collection.immutable.Seq<<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T>> rest, |
| scala.reflect.ClassTag<T> evidence$7)</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Build the union of a list of RDDs passed as variable-length arguments.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><T> <a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#union(scala.collection.immutable.Seq,scala.reflect.ClassTag)" class="member-name-link">union</a><wbr>(scala.collection.immutable.Seq<<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T>> rdds, |
| scala.reflect.ClassTag<T> evidence$6)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Build the union of a list of RDDs.</div> |
| </div> |
| <div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></code></div> |
| <div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#version()" class="member-name-link">version</a>()</code></div> |
| <div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">The version of Spark on which this application is running.</div> |
| </div> |
| <div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>>></code></div> |
| <div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"><code><a href="#wholeTextFiles(java.lang.String,int)" class="member-name-link">wholeTextFiles</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions)</code></div> |
| <div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab4"> |
| <div class="block">Read a directory of text files from HDFS, a local file system (available on all nodes), or any |
| Hadoop-supported file system URI.</div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="inherited-list"> |
| <h3 id="methods-inherited-from-class-Object">Methods inherited from class java.lang.<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></h3> |
| <code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#equals(java.lang.Object)" title="class or interface in java.lang" class="external-link">equals</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#getClass()" title="class or interface in java.lang" class="external-link">getClass</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#hashCode()" title="class or interface in java.lang" class="external-link">hashCode</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#notify()" title="class or interface in java.lang" class="external-link">notify</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#notifyAll()" title="class or interface in java.lang" class="external-link">notifyAll</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#toString()" title="class or interface in java.lang" class="external-link">toString</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#wait()" title="class or interface in java.lang" class="external-link">wait</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#wait(long)" title="class or interface in java.lang" class="external-link">wait</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#wait(long,int)" title="class or interface in java.lang" class="external-link">wait</a></code></div> |
| <div class="inherited-list"> |
| <h3 id="methods-inherited-from-class-org.apache.spark.internal.Logging">Methods inherited from interface org.apache.spark.internal.Logging</h3> |
| <code>initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext</code></div> |
| </section> |
| </li> |
| </ul> |
| </section> |
| <section class="details"> |
| <ul class="details-list"> |
| <!-- ========= CONSTRUCTOR DETAIL ======== --> |
| <li> |
| <section class="constructor-details" id="constructor-detail"> |
| <h2>Constructor Details</h2> |
| <ul class="member-list"> |
| <li> |
| <section class="detail" id="<init>(org.apache.spark.SparkConf)"> |
| <h3>SparkContext</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="element-name">SparkContext</span><wbr><span class="parameters">(<a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a> config)</span></div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="<init>()"> |
| <h3>SparkContext</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="element-name">SparkContext</span>()</div> |
| <div class="block">Create a SparkContext that loads settings from system properties (for instance, when |
| launching with ./bin/spark-submit).</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="<init>(java.lang.String,java.lang.String,org.apache.spark.SparkConf)"> |
| <h3>SparkContext</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="element-name">SparkContext</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> master, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> appName, |
| <a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a> conf)</span></div> |
| <div class="block">Alternative constructor that allows setting common Spark properties directly |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>master</code> - Cluster URL to connect to (e.g. spark://host:port, local[4]).</dd> |
| <dd><code>appName</code> - A name for your application, to display on the cluster web UI</dd> |
| <dd><code>conf</code> - a <a href="SparkConf.html" title="class in org.apache.spark"><code>SparkConf</code></a> object specifying other Spark parameters</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="<init>(java.lang.String,java.lang.String,java.lang.String,scala.collection.immutable.Seq,scala.collection.Map)"> |
| <h3>SparkContext</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="element-name">SparkContext</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> master, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> appName, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> sparkHome, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>> jars, |
| scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>> environment)</span></div> |
| <div class="block">Alternative constructor that allows setting common Spark properties directly |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>master</code> - Cluster URL to connect to (e.g. spark://host:port, local[4]).</dd> |
| <dd><code>appName</code> - A name for your application, to display on the cluster web UI.</dd> |
| <dd><code>sparkHome</code> - Location where Spark is installed on cluster nodes.</dd> |
| <dd><code>jars</code> - Collection of JARs to send to the cluster. These can be paths on the local file |
| system or HDFS, HTTP, HTTPS, or FTP URLs.</dd> |
| <dd><code>environment</code> - Environment variables to set on worker nodes.</dd> |
| </dl> |
| </section> |
| </li> |
| </ul> |
| </section> |
| </li> |
| <!-- ============ METHOD DETAIL ========== --> |
| <li> |
| <section class="method-details" id="method-detail"> |
| <h2>Method Details</h2> |
| <ul class="member-list"> |
| <li> |
| <section class="detail" id="getOrCreate(org.apache.spark.SparkConf)"> |
| <h3>getOrCreate</h3> |
| <div class="member-signature"><span class="modifiers">public static</span> <span class="return-type"><a href="SparkContext.html" title="class in org.apache.spark">SparkContext</a></span> <span class="element-name">getOrCreate</span><wbr><span class="parameters">(<a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a> config)</span></div> |
| <div class="block">This function may be used to get or instantiate a SparkContext and register it as a |
| singleton object. Because we can only have one active SparkContext per JVM, |
| this is useful when applications may wish to share a SparkContext. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>config</code> - <code>SparkConfig</code> that will be used for initialisation of the <code>SparkContext</code></dd> |
| <dt>Returns:</dt> |
| <dd>current <code>SparkContext</code> (or a new one if it wasn't created before the function call)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getOrCreate()"> |
| <h3>getOrCreate</h3> |
| <div class="member-signature"><span class="modifiers">public static</span> <span class="return-type"><a href="SparkContext.html" title="class in org.apache.spark">SparkContext</a></span> <span class="element-name">getOrCreate</span>()</div> |
| <div class="block">This function may be used to get or instantiate a SparkContext and register it as a |
| singleton object. Because we can only have one active SparkContext per JVM, |
| this is useful when applications may wish to share a SparkContext. |
| <p> |
| This method allows not passing a SparkConf (useful if just retrieving). |
| <p></div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>current <code>SparkContext</code> (or a new one if wasn't created before the function call)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="jarOfClass(java.lang.Class)"> |
| <h3>jarOfClass</h3> |
| <div class="member-signature"><span class="modifiers">public static</span> <span class="return-type">scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">jarOfClass</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><?> cls)</span></div> |
| <div class="block">Find the JAR from which a given class was loaded, to make it easy for users to pass |
| their JARs to SparkContext. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>cls</code> - class that should be inside of the jar</dd> |
| <dt>Returns:</dt> |
| <dd>jar that contains the Class, <code>None</code> if not found</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="jarOfObject(java.lang.Object)"> |
| <h3>jarOfObject</h3> |
| <div class="member-signature"><span class="modifiers">public static</span> <span class="return-type">scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">jarOfObject</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a> obj)</span></div> |
| <div class="block">Find the JAR that contains the class of a particular object, to make it easy for users |
| to pass their JARs to SparkContext. In most cases you can call jarOfObject(this) in |
| your driver program. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>obj</code> - reference to an instance which class should be inside of the jar</dd> |
| <dt>Returns:</dt> |
| <dd>jar that contains the class of the instance, <code>None</code> if not found</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="org$apache$spark$internal$Logging$$log_()"> |
| <h3>org$apache$spark$internal$Logging$$log_</h3> |
| <div class="member-signature"><span class="modifiers">public static</span> <span class="return-type">org.slf4j.Logger</span> <span class="element-name">org$apache$spark$internal$Logging$$log_</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger)"> |
| <h3>org$apache$spark$internal$Logging$$log__$eq</h3> |
| <div class="member-signature"><span class="modifiers">public static</span> <span class="return-type">void</span> <span class="element-name">org$apache$spark$internal$Logging$$log__$eq</span><wbr><span class="parameters">(org.slf4j.Logger x$1)</span></div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="LogStringContext(scala.StringContext)"> |
| <h3>LogStringContext</h3> |
| <div class="member-signature"><span class="modifiers">public static</span> <span class="return-type">org.apache.spark.internal.Logging.LogStringContext</span> <span class="element-name">LogStringContext</span><wbr><span class="parameters">(scala.StringContext sc)</span></div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="startTime()"> |
| <h3>startTime</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">long</span> <span class="element-name">startTime</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getConf()"> |
| <h3>getConf</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="SparkConf.html" title="class in org.apache.spark">SparkConf</a></span> <span class="element-name">getConf</span>()</div> |
| <div class="block">Return a copy of this SparkContext's configuration. The configuration ''cannot'' be |
| changed at runtime.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="resources()"> |
| <h3>resources</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="resource/ResourceInformation.html" title="class in org.apache.spark.resource">ResourceInformation</a>></span> <span class="element-name">resources</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="jars()"> |
| <h3>jars</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">jars</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="files()"> |
| <h3>files</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">files</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="archives()"> |
| <h3>archives</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">archives</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="master()"> |
| <h3>master</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></span> <span class="element-name">master</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="deployMode()"> |
| <h3>deployMode</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></span> <span class="element-name">deployMode</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="appName()"> |
| <h3>appName</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></span> <span class="element-name">appName</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="isLocal()"> |
| <h3>isLocal</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">boolean</span> <span class="element-name">isLocal</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="isStopped()"> |
| <h3>isStopped</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">boolean</span> <span class="element-name">isStopped</span>()</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>true if context is stopped or in the midst of stopping.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="statusTracker()"> |
| <h3>statusTracker</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="SparkStatusTracker.html" title="class in org.apache.spark">SparkStatusTracker</a></span> <span class="element-name">statusTracker</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="uiWebUrl()"> |
| <h3>uiWebUrl</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">uiWebUrl</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="hadoopConfiguration()"> |
| <h3>hadoopConfiguration</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">org.apache.hadoop.conf.Configuration</span> <span class="element-name">hadoopConfiguration</span>()</div> |
| <div class="block">A default Hadoop Configuration for the Hadoop code (e.g. file systems) that we reuse. |
| <p></div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| <dt>Note:</dt> |
| <dd>As it will be reused in all Hadoop RDDs, it's better not to modify it unless you |
| plan to set some global configurations for all Hadoop RDDs.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="sparkUser()"> |
| <h3>sparkUser</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></span> <span class="element-name">sparkUser</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="applicationId()"> |
| <h3>applicationId</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></span> <span class="element-name">applicationId</span>()</div> |
| <div class="block">A unique identifier for the Spark application. |
| Its format depends on the scheduler implementation. |
| (i.e. |
| in case of local spark app something like 'local-1433865536131' |
| in case of YARN something like 'application_1433865536131_34483' |
| )</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="applicationAttemptId()"> |
| <h3>applicationAttemptId</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">applicationAttemptId</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="setLogLevel(java.lang.String)"> |
| <h3>setLogLevel</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">setLogLevel</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> logLevel)</span></div> |
| <div class="block">Control our logLevel. This overrides any user-defined log settings.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>logLevel</code> - The desired log level as a string. |
| Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="setLocalProperty(java.lang.String,java.lang.String)"> |
| <h3>setLocalProperty</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">setLocalProperty</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> key, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> value)</span></div> |
| <div class="block">Set a local property that affects jobs submitted from this thread, such as the Spark fair |
| scheduler pool. User-defined properties may also be set here. These properties are propagated |
| through to worker tasks and can be accessed there via |
| <a href="TaskContext.html#getLocalProperty(java.lang.String)"><code>TaskContext.getLocalProperty(java.lang.String)</code></a>. |
| <p> |
| These properties are inherited by child threads spawned from this thread. This |
| may have unexpected consequences when working with thread pools. The standard java |
| implementation of thread pools have worker threads spawn other worker threads. |
| As a result, local properties may propagate unpredictably. |
| <p> |
| To remove/unset property simply set <code>value</code> to null e.g. sc.setLocalProperty("key", null)</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>key</code> - (undocumented)</dd> |
| <dd><code>value</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getLocalProperty(java.lang.String)"> |
| <h3>getLocalProperty</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></span> <span class="element-name">getLocalProperty</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> key)</span></div> |
| <div class="block">Get a local property set in this thread, or null if it is missing. See |
| <code>org.apache.spark.SparkContext.setLocalProperty</code>.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>key</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="setJobDescription(java.lang.String)"> |
| <h3>setJobDescription</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">setJobDescription</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> value)</span></div> |
| <div class="block">Set a human readable description of the current job.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="setJobGroup(java.lang.String,java.lang.String,boolean)"> |
| <h3>setJobGroup</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">setJobGroup</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> groupId, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> description, |
| boolean interruptOnCancel)</span></div> |
| <div class="block">Assigns a group ID to all the jobs started by this thread until the group ID is set to a |
| different value or cleared. |
| <p> |
| Often, a unit of execution in an application consists of multiple Spark actions or jobs. |
| Application programmers can use this method to group all those jobs together and give a |
| group description. Once set, the Spark web UI will associate such jobs with this group. |
| <p> |
| The application can also use <code>org.apache.spark.SparkContext.cancelJobGroup</code> to cancel all |
| running jobs in this group. For example, |
| <pre><code> |
| // In the main thread: |
| sc.setJobGroup("some_job_to_cancel", "some job description") |
| sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count() |
| |
| // In a separate thread: |
| sc.cancelJobGroup("some_job_to_cancel") |
| </code></pre> |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>interruptOnCancel</code> - If true, then job cancellation will result in <code>Thread.interrupt()</code> |
| being called on the job's executor threads. This is useful to help ensure that the tasks |
| are actually stopped in a timely manner, but is off by default due to HDFS-1208, where HDFS |
| may respond to Thread.interrupt() by marking nodes as dead.</dd> |
| <dd><code>groupId</code> - (undocumented)</dd> |
| <dd><code>description</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="clearJobGroup()"> |
| <h3>clearJobGroup</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">clearJobGroup</span>()</div> |
| <div class="block">Clear the current thread's job group ID and its description.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="setInterruptOnCancel(boolean)"> |
| <h3>setInterruptOnCancel</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">setInterruptOnCancel</span><wbr><span class="parameters">(boolean interruptOnCancel)</span></div> |
| <div class="block">Set the behavior of job cancellation from jobs started in this thread. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>interruptOnCancel</code> - If true, then job cancellation will result in <code>Thread.interrupt()</code> |
| being called on the job's executor threads. This is useful to help ensure that the tasks |
| are actually stopped in a timely manner, but is off by default due to HDFS-1208, where HDFS |
| may respond to Thread.interrupt() by marking nodes as dead. |
| <p></dd> |
| <dt>Since:</dt> |
| <dd>3.5.0</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="addJobTag(java.lang.String)"> |
| <h3>addJobTag</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">addJobTag</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> tag)</span></div> |
| <div class="block">Add a tag to be assigned to all the jobs started by this thread. |
| <p> |
| Often, a unit of execution in an application consists of multiple Spark actions or jobs. |
| Application programmers can use this method to group all those jobs together and give a |
| group tag. The application can use <code>org.apache.spark.sql.SparkSession.interruptTag</code> to cancel |
| all running executions with this tag. For example: |
| <pre><code> |
| // In the main thread: |
| sc.addJobTag("myjobs") |
| sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count() |
| |
| // In a separate thread: |
| spark.cancelJobsWithTag("myjobs") |
| </code></pre> |
| <p> |
| There may be multiple tags present at the same time, so different parts of application may use |
| different tags to perform cancellation at different levels of granularity. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>tag</code> - The tag to be added. Cannot contain ',' (comma) character. |
| <p></dd> |
| <dt>Since:</dt> |
| <dd>3.5.0</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="removeJobTag(java.lang.String)"> |
| <h3>removeJobTag</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">removeJobTag</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> tag)</span></div> |
| <div class="block">Remove a tag previously added to be assigned to all the jobs started by this thread. |
| Noop if such a tag was not added earlier. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>tag</code> - The tag to be removed. Cannot contain ',' (comma) character. |
| <p></dd> |
| <dt>Since:</dt> |
| <dd>3.5.0</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getJobTags()"> |
| <h3>getJobTags</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Set<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">getJobTags</span>()</div> |
| <div class="block">Get the tags that are currently set to be assigned to all the jobs started by this thread. |
| <p></div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| <dt>Since:</dt> |
| <dd>3.5.0</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="clearJobTags()"> |
| <h3>clearJobTags</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">clearJobTags</span>()</div> |
| <div class="block">Clear the current thread's job tags. |
| <p></div> |
| <dl class="notes"> |
| <dt>Since:</dt> |
| <dd>3.5.0</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="parallelize(scala.collection.immutable.Seq,int,scala.reflect.ClassTag)"> |
| <h3>parallelize</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></span> <span class="element-name">parallelize</span><wbr><span class="parameters">(scala.collection.immutable.Seq<T> seq, |
| int numSlices, |
| scala.reflect.ClassTag<T> evidence$1)</span></div> |
| <div class="block">Distribute a local Scala collection to form an RDD. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>seq</code> - Scala collection to distribute</dd> |
| <dd><code>numSlices</code> - number of partitions to divide the collection into</dd> |
| <dd><code>evidence$1</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD representing distributed collection</dd> |
| <dt>Note:</dt> |
| <dd>Parallelize acts lazily. If <code>seq</code> is a mutable collection and is altered after the call |
| to parallelize and before the first action on the RDD, the resultant RDD will reflect the |
| modified collection. Pass a copy of the argument to avoid this., avoid using <code>parallelize(Seq())</code> to create an empty <code>RDD</code>. Consider <code>emptyRDD</code> for an |
| RDD with no partitions, or <code>parallelize(Seq[T]())</code> for an RDD of <code>T</code> with empty partitions.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="range(long,long,long,int)"> |
| <h3>range</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>></span> <span class="element-name">range</span><wbr><span class="parameters">(long start, |
| long end, |
| long step, |
| int numSlices)</span></div> |
| <div class="block">Creates a new RDD[Long] containing elements from <code>start</code> to <code>end</code>(exclusive), increased by |
| <code>step</code> every element. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>start</code> - the start value.</dd> |
| <dd><code>end</code> - the end value.</dd> |
| <dd><code>step</code> - the incremental step</dd> |
| <dd><code>numSlices</code> - number of partitions to divide the collection into</dd> |
| <dt>Returns:</dt> |
| <dd>RDD representing distributed range</dd> |
| <dt>Note:</dt> |
| <dd>if we need to cache this RDD, we should make sure each partition does not exceed limit. |
| <p></dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="makeRDD(scala.collection.immutable.Seq,int,scala.reflect.ClassTag)"> |
| <h3>makeRDD</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></span> <span class="element-name">makeRDD</span><wbr><span class="parameters">(scala.collection.immutable.Seq<T> seq, |
| int numSlices, |
| scala.reflect.ClassTag<T> evidence$2)</span></div> |
| <div class="block">Distribute a local Scala collection to form an RDD. |
| <p> |
| This method is identical to <code>parallelize</code>.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>seq</code> - Scala collection to distribute</dd> |
| <dd><code>numSlices</code> - number of partitions to divide the collection into</dd> |
| <dd><code>evidence$2</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD representing distributed collection</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="makeRDD(scala.collection.immutable.Seq,scala.reflect.ClassTag)"> |
| <h3>makeRDD</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></span> <span class="element-name">makeRDD</span><wbr><span class="parameters">(scala.collection.immutable.Seq<scala.Tuple2<T,<wbr>scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>>>> seq, |
| scala.reflect.ClassTag<T> evidence$3)</span></div> |
| <div class="block">Distribute a local Scala collection to form an RDD, with one or more |
| location preferences (hostnames of Spark nodes) for each object. |
| Create a new partition for each collection item.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>seq</code> - list of tuples of data and location preferences (hostnames of Spark nodes)</dd> |
| <dd><code>evidence$3</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD representing data partitioned according to location preferences</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="textFile(java.lang.String,int)"> |
| <h3>textFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">textFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions)</span></div> |
| <div class="block">Read a text file from HDFS, a local file system (available on all nodes), or any |
| Hadoop-supported file system URI, and return it as an RDD of Strings. |
| The text files must be encoded as UTF-8. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - path to the text file on a supported file system</dd> |
| <dd><code>minPartitions</code> - suggested minimum number of partitions for the resulting RDD</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of lines of the text file</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="wholeTextFiles(java.lang.String,int)"> |
| <h3>wholeTextFiles</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>>></span> <span class="element-name">wholeTextFiles</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions)</span></div> |
| <div class="block">Read a directory of text files from HDFS, a local file system (available on all nodes), or any |
| Hadoop-supported file system URI. Each file is read as a single record and returned in a |
| key-value pair, where the key is the path of each file, the value is the content of each file. |
| The text files must be encoded as UTF-8. |
| <p> |
| <p> For example, if you have the following files: |
| <pre><code> |
| hdfs://a-hdfs-path/part-00000 |
| hdfs://a-hdfs-path/part-00001 |
| ... |
| hdfs://a-hdfs-path/part-nnnnn |
| </code></pre> |
| <p> |
| Do <code>val rdd = sparkContext.wholeTextFile("hdfs://a-hdfs-path")</code>, |
| <p> |
| <p> then <code>rdd</code> contains |
| <pre><code> |
| (a-hdfs-path/part-00000, its content) |
| (a-hdfs-path/part-00001, its content) |
| ... |
| (a-hdfs-path/part-nnnnn, its content) |
| </code></pre> |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - Directory to the input data files, the path can be comma separated paths as the |
| list of inputs.</dd> |
| <dd><code>minPartitions</code> - A suggestion value of the minimal splitting number for input data.</dd> |
| <dt>Returns:</dt> |
| <dd>RDD representing tuples of file path and the corresponding file content</dd> |
| <dt>Note:</dt> |
| <dd>Small files are preferred, large file is also allowable, but may cause bad performance., On some filesystems, <code>.../path/&#42;</code> can be a more efficient way to read all files |
| in a directory rather than <code>.../path/</code> or <code>.../path</code>, Partitioning is determined by data locality. This may result in too few partitions |
| by default. |
| <p></dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="binaryFiles(java.lang.String,int)"> |
| <h3>binaryFiles</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="input/PortableDataStream.html" title="class in org.apache.spark.input">PortableDataStream</a>>></span> <span class="element-name">binaryFiles</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions)</span></div> |
| <div class="block">Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file |
| (useful for binary data) |
| <p> |
| For example, if you have the following files: |
| <pre><code> |
| hdfs://a-hdfs-path/part-00000 |
| hdfs://a-hdfs-path/part-00001 |
| ... |
| hdfs://a-hdfs-path/part-nnnnn |
| </code></pre> |
| <p> |
| Do |
| <code>val rdd = sparkContext.binaryFiles("hdfs://a-hdfs-path")</code>, |
| <p> |
| then <code>rdd</code> contains |
| <pre><code> |
| (a-hdfs-path/part-00000, its content) |
| (a-hdfs-path/part-00001, its content) |
| ... |
| (a-hdfs-path/part-nnnnn, its content) |
| </code></pre> |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - Directory to the input data files, the path can be comma separated paths as the |
| list of inputs.</dd> |
| <dd><code>minPartitions</code> - A suggestion value of the minimal splitting number for input data.</dd> |
| <dt>Returns:</dt> |
| <dd>RDD representing tuples of file path and corresponding file content</dd> |
| <dt>Note:</dt> |
| <dd>Small files are preferred; very large files may cause bad performance., On some filesystems, <code>.../path/&#42;</code> can be a more efficient way to read all files |
| in a directory rather than <code>.../path/</code> or <code>.../path</code>, Partitioning is determined by data locality. This may result in too few partitions |
| by default. |
| <p></dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="binaryRecords(java.lang.String,int,org.apache.hadoop.conf.Configuration)"> |
| <h3>binaryRecords</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><byte[]></span> <span class="element-name">binaryRecords</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int recordLength, |
| org.apache.hadoop.conf.Configuration conf)</span></div> |
| <div class="block">Load data from a flat binary file, assuming the length of each record is constant. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - Directory to the input data files, the path can be comma separated paths as the |
| list of inputs.</dd> |
| <dd><code>recordLength</code> - The length at which to split the records</dd> |
| <dd><code>conf</code> - Configuration for setting up the dataset. |
| <p></dd> |
| <dt>Returns:</dt> |
| <dd>An RDD of data with values, represented as byte arrays</dd> |
| <dt>Note:</dt> |
| <dd>We ensure that the byte array for each record in the resulting RDD |
| has the provided record length. |
| <p></dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="hadoopRDD(org.apache.hadoop.mapred.JobConf,java.lang.Class,java.lang.Class,java.lang.Class,int)"> |
| <h3>hadoopRDD</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><K,<wbr> |
| V></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">hadoopRDD</span><wbr><span class="parameters">(org.apache.hadoop.mapred.JobConf conf, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><? extends org.apache.hadoop.mapred.InputFormat<K,<wbr>V>> inputFormatClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass, |
| int minPartitions)</span></div> |
| <div class="block">Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other |
| necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable), |
| using the older MapReduce API (<code>org.apache.hadoop.mapred</code>). |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>conf</code> - JobConf for setting up the dataset. Note: This will be put into a Broadcast. |
| Therefore if you plan to reuse this conf to create multiple RDDs, you need to make |
| sure you won't modify the conf. A safe approach is always creating a new conf for |
| a new RDD.</dd> |
| <dd><code>inputFormatClass</code> - storage format of the data to be read</dd> |
| <dd><code>keyClass</code> - <code>Class</code> of the key associated with the <code>inputFormatClass</code> parameter</dd> |
| <dd><code>valueClass</code> - <code>Class</code> of the value associated with the <code>inputFormatClass</code> parameter</dd> |
| <dd><code>minPartitions</code> - Minimum number of Hadoop Splits to generate.</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value |
| <p></dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="hadoopFile(java.lang.String,java.lang.Class,java.lang.Class,java.lang.Class,int)"> |
| <h3>hadoopFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><K,<wbr> |
| V></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">hadoopFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><? extends org.apache.hadoop.mapred.InputFormat<K,<wbr>V>> inputFormatClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass, |
| int minPartitions)</span></div> |
| <div class="block">Get an RDD for a Hadoop file with an arbitrary InputFormat |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>inputFormatClass</code> - storage format of the data to be read</dd> |
| <dd><code>keyClass</code> - <code>Class</code> of the key associated with the <code>inputFormatClass</code> parameter</dd> |
| <dd><code>valueClass</code> - <code>Class</code> of the value associated with the <code>inputFormatClass</code> parameter</dd> |
| <dd><code>minPartitions</code> - suggested minimum number of partitions for the resulting RDD</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="hadoopFile(java.lang.String,int,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.reflect.ClassTag)"> |
| <h3>hadoopFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters-long"><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapred.InputFormat<K,<wbr> |
| V>></span> |
| <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">hadoopFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.reflect.ClassTag<F> fm)</span></div> |
| <div class="block">Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, |
| values and the InputFormat so that users don't need to pass them directly. Instead, callers |
| can just write, for example, |
| <pre><code> |
| val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path, minPartitions) |
| </code></pre> |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>minPartitions</code> - suggested minimum number of partitions for the resulting RDD</dd> |
| <dd><code>km</code> - (undocumented)</dd> |
| <dd><code>vm</code> - (undocumented)</dd> |
| <dd><code>fm</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="hadoopFile(java.lang.String,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.reflect.ClassTag)"> |
| <h3>hadoopFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters-long"><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapred.InputFormat<K,<wbr> |
| V>></span> |
| <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">hadoopFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.reflect.ClassTag<F> fm)</span></div> |
| <div class="block">Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, |
| values and the InputFormat so that users don't need to pass them directly. Instead, callers |
| can just write, for example, |
| <pre><code> |
| val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path) |
| </code></pre> |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths as |
| a list of inputs</dd> |
| <dd><code>km</code> - (undocumented)</dd> |
| <dd><code>vm</code> - (undocumented)</dd> |
| <dd><code>fm</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="newAPIHadoopFile(java.lang.String,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.reflect.ClassTag)"> |
| <h3>newAPIHadoopFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters-long"><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapreduce.InputFormat<K,<wbr> |
| V>></span> |
| <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">newAPIHadoopFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.reflect.ClassTag<F> fm)</span></div> |
| <div class="block">Smarter version of <code>newApiHadoopFile</code> that uses class tags to figure out the classes of keys, |
| values and the <code>org.apache.hadoop.mapreduce.InputFormat</code> (new MapReduce API) so that user |
| don't need to pass them directly. Instead, callers can just write, for example: |
| <code></code><code> |
| val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path) |
| </code><code></code> |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>km</code> - (undocumented)</dd> |
| <dd><code>vm</code> - (undocumented)</dd> |
| <dd><code>fm</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="newAPIHadoopFile(java.lang.String,java.lang.Class,java.lang.Class,java.lang.Class,org.apache.hadoop.conf.Configuration)"> |
| <h3>newAPIHadoopFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters-long"><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapreduce.InputFormat<K,<wbr> |
| V>></span> |
| <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">newAPIHadoopFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><F> fClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> kClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> vClass, |
| org.apache.hadoop.conf.Configuration conf)</span></div> |
| <div class="block">Get an RDD for a given Hadoop file with an arbitrary new API InputFormat |
| and extra configuration options to pass to the input format. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>fClass</code> - storage format of the data to be read</dd> |
| <dd><code>kClass</code> - <code>Class</code> of the key associated with the <code>fClass</code> parameter</dd> |
| <dd><code>vClass</code> - <code>Class</code> of the value associated with the <code>fClass</code> parameter</dd> |
| <dd><code>conf</code> - Hadoop configuration</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="newAPIHadoopRDD(org.apache.hadoop.conf.Configuration,java.lang.Class,java.lang.Class,java.lang.Class)"> |
| <h3>newAPIHadoopRDD</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters-long"><K,<wbr> |
| V,<wbr> |
| F extends org.apache.hadoop.mapreduce.InputFormat<K,<wbr> |
| V>></span> |
| <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">newAPIHadoopRDD</span><wbr><span class="parameters">(org.apache.hadoop.conf.Configuration conf, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><F> fClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> kClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> vClass)</span></div> |
| <div class="block">Get an RDD for a given Hadoop file with an arbitrary new API InputFormat |
| and extra configuration options to pass to the input format. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>conf</code> - Configuration for setting up the dataset. Note: This will be put into a Broadcast. |
| Therefore if you plan to reuse this conf to create multiple RDDs, you need to make |
| sure you won't modify the conf. A safe approach is always creating a new conf for |
| a new RDD.</dd> |
| <dd><code>fClass</code> - storage format of the data to be read</dd> |
| <dd><code>kClass</code> - <code>Class</code> of the key associated with the <code>fClass</code> parameter</dd> |
| <dd><code>vClass</code> - <code>Class</code> of the value associated with the <code>fClass</code> parameter |
| <p></dd> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="sequenceFile(java.lang.String,java.lang.Class,java.lang.Class,int)"> |
| <h3>sequenceFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><K,<wbr> |
| V></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">sequenceFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass, |
| int minPartitions)</span></div> |
| <div class="block">Get an RDD for a Hadoop SequenceFile with given key and value types. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>keyClass</code> - <code>Class</code> of the key associated with <code>SequenceFileInputFormat</code></dd> |
| <dd><code>valueClass</code> - <code>Class</code> of the value associated with <code>SequenceFileInputFormat</code></dd> |
| <dd><code>minPartitions</code> - suggested minimum number of partitions for the resulting RDD</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="sequenceFile(java.lang.String,java.lang.Class,java.lang.Class)"> |
| <h3>sequenceFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><K,<wbr> |
| V></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">sequenceFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><K> keyClass, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Class.html" title="class or interface in java.lang" class="external-link">Class</a><V> valueClass)</span></div> |
| <div class="block">Get an RDD for a Hadoop SequenceFile with given key and value types. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>keyClass</code> - <code>Class</code> of the key associated with <code>SequenceFileInputFormat</code></dd> |
| <dd><code>valueClass</code> - <code>Class</code> of the value associated with <code>SequenceFileInputFormat</code></dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="sequenceFile(java.lang.String,int,scala.reflect.ClassTag,scala.reflect.ClassTag,scala.Function0,scala.Function0)"> |
| <h3>sequenceFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><K,<wbr> |
| V></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><scala.Tuple2<K,<wbr>V>></span> <span class="element-name">sequenceFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions, |
| scala.reflect.ClassTag<K> km, |
| scala.reflect.ClassTag<V> vm, |
| scala.Function0<org.apache.spark.WritableConverter<K>> kcf, |
| scala.Function0<org.apache.spark.WritableConverter<V>> vcf)</span></div> |
| <div class="block">Version of sequenceFile() for types implicitly convertible to Writables through a |
| WritableConverter. For example, to access a SequenceFile where the keys are Text and the |
| values are IntWritable, you could simply write |
| <pre><code> |
| sparkContext.sequenceFile[String, Int](path, ...) |
| </code></pre> |
| <p> |
| WritableConverters are provided in a somewhat strange way (by an implicit function) to support |
| both subclasses of Writable and types for which we define a converter (e.g. Int to |
| IntWritable). The most natural thing would've been to have implicit objects for the |
| converters, but then we couldn't have an object for every subclass of Writable (you can't |
| have a parameterized singleton object). We use functions instead to create a new converter |
| for the appropriate type. In addition, we pass the converter a ClassTag of its type to |
| allow it to figure out the Writable class to use in the subclass case. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>minPartitions</code> - suggested minimum number of partitions for the resulting RDD</dd> |
| <dd><code>km</code> - (undocumented)</dd> |
| <dd><code>vm</code> - (undocumented)</dd> |
| <dd><code>kcf</code> - (undocumented)</dd> |
| <dd><code>vcf</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD of tuples of key and corresponding value</dd> |
| <dt>Note:</dt> |
| <dd>Because Hadoop's RecordReader class re-uses the same Writable object for each |
| record, directly caching the returned RDD or directly passing it to an aggregation or shuffle |
| operation will create many references to the same object. |
| If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first |
| copy them using a <code>map</code> function.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="objectFile(java.lang.String,int,scala.reflect.ClassTag)"> |
| <h3>objectFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></span> <span class="element-name">objectFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| int minPartitions, |
| scala.reflect.ClassTag<T> evidence$4)</span></div> |
| <div class="block">Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and |
| BytesWritable values that contain a serialized partition. This is still an experimental |
| storage format and may not be supported exactly as is in future Spark releases. It will also |
| be pretty slow if you use the default serializer (Java serialization), |
| though the nice thing about it is that there's very little effort required to save arbitrary |
| objects. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - directory to the input data files, the path can be comma separated paths |
| as a list of inputs</dd> |
| <dd><code>minPartitions</code> - suggested minimum number of partitions for the resulting RDD</dd> |
| <dd><code>evidence$4</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>RDD representing deserialized data from the file(s)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="union(scala.collection.immutable.Seq,scala.reflect.ClassTag)"> |
| <h3>union</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></span> <span class="element-name">union</span><wbr><span class="parameters">(scala.collection.immutable.Seq<<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T>> rdds, |
| scala.reflect.ClassTag<T> evidence$6)</span></div> |
| <div class="block">Build the union of a list of RDDs.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="union(org.apache.spark.rdd.RDD,scala.collection.immutable.Seq,scala.reflect.ClassTag)"> |
| <h3>union</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></span> <span class="element-name">union</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> first, |
| scala.collection.immutable.Seq<<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T>> rest, |
| scala.reflect.ClassTag<T> evidence$7)</span></div> |
| <div class="block">Build the union of a list of RDDs passed as variable-length arguments.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="emptyRDD(scala.reflect.ClassTag)"> |
| <h3>emptyRDD</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T></span> <span class="element-name">emptyRDD</span><wbr><span class="parameters">(scala.reflect.ClassTag<T> evidence$8)</span></div> |
| <div class="block">Get an RDD that has no partitions or elements.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="register(org.apache.spark.util.AccumulatorV2)"> |
| <h3>register</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">register</span><wbr><span class="parameters">(<a href="util/AccumulatorV2.html" title="class in org.apache.spark.util">AccumulatorV2</a><?,<wbr>?> acc)</span></div> |
| <div class="block">Register the given accumulator. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>acc</code> - (undocumented)</dd> |
| <dt>Note:</dt> |
| <dd>Accumulators must be registered before use, or it will throw exception.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="register(org.apache.spark.util.AccumulatorV2,java.lang.String)"> |
| <h3>register</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">register</span><wbr><span class="parameters">(<a href="util/AccumulatorV2.html" title="class in org.apache.spark.util">AccumulatorV2</a><?,<wbr>?> acc, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</span></div> |
| <div class="block">Register the given accumulator with given name. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>acc</code> - (undocumented)</dd> |
| <dd><code>name</code> - (undocumented)</dd> |
| <dt>Note:</dt> |
| <dd>Accumulators must be registered before use, or it will throw exception.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="longAccumulator()"> |
| <h3>longAccumulator</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="util/LongAccumulator.html" title="class in org.apache.spark.util">LongAccumulator</a></span> <span class="element-name">longAccumulator</span>()</div> |
| <div class="block">Create and register a long accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="longAccumulator(java.lang.String)"> |
| <h3>longAccumulator</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="util/LongAccumulator.html" title="class in org.apache.spark.util">LongAccumulator</a></span> <span class="element-name">longAccumulator</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</span></div> |
| <div class="block">Create and register a long accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>name</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="doubleAccumulator()"> |
| <h3>doubleAccumulator</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="util/DoubleAccumulator.html" title="class in org.apache.spark.util">DoubleAccumulator</a></span> <span class="element-name">doubleAccumulator</span>()</div> |
| <div class="block">Create and register a double accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="doubleAccumulator(java.lang.String)"> |
| <h3>doubleAccumulator</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="util/DoubleAccumulator.html" title="class in org.apache.spark.util">DoubleAccumulator</a></span> <span class="element-name">doubleAccumulator</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</span></div> |
| <div class="block">Create and register a double accumulator, which starts with 0 and accumulates inputs by <code>add</code>.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>name</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="collectionAccumulator()"> |
| <h3>collectionAccumulator</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="util/CollectionAccumulator.html" title="class in org.apache.spark.util">CollectionAccumulator</a><T></span> <span class="element-name">collectionAccumulator</span>()</div> |
| <div class="block">Create and register a <code>CollectionAccumulator</code>, which starts with empty list and accumulates |
| inputs by adding them into the list.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="collectionAccumulator(java.lang.String)"> |
| <h3>collectionAccumulator</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="util/CollectionAccumulator.html" title="class in org.apache.spark.util">CollectionAccumulator</a><T></span> <span class="element-name">collectionAccumulator</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> name)</span></div> |
| <div class="block">Create and register a <code>CollectionAccumulator</code>, which starts with empty list and accumulates |
| inputs by adding them into the list.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>name</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="broadcast(T,scala.reflect.ClassTag)"> |
| <h3 id="broadcast(java.lang.Object,scala.reflect.ClassTag)">broadcast</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T></span> <span class="return-type"><a href="broadcast/Broadcast.html" title="class in org.apache.spark.broadcast">Broadcast</a><T></span> <span class="element-name">broadcast</span><wbr><span class="parameters">(T value, |
| scala.reflect.ClassTag<T> evidence$9)</span></div> |
| <div class="block">Broadcast a read-only variable to the cluster, returning a |
| <a href="broadcast/Broadcast.html" title="class in org.apache.spark.broadcast"><code>Broadcast</code></a> object for reading it in distributed functions. |
| The variable will be sent to each executor only once. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>value</code> - value to broadcast to the Spark nodes</dd> |
| <dd><code>evidence$9</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd><code>Broadcast</code> object, a read-only variable cached on each machine</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="addFile(java.lang.String)"> |
| <h3>addFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">addFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path)</span></div> |
| <div class="block">Add a file to be downloaded with this Spark job on every node. |
| <p> |
| If a file is added during execution, it will not be available until the next TaskSet starts. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - can be either a local file, a file in HDFS (or other Hadoop-supported |
| filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, |
| use <code>SparkFiles.get(fileName)</code> to find its download location. |
| <p></dd> |
| <dt>Note:</dt> |
| <dd>A path can be added only once. Subsequent additions of the same path are ignored.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="listFiles()"> |
| <h3>listFiles</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">listFiles</span>()</div> |
| <div class="block">Returns a list of file paths that are added to resources.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="addArchive(java.lang.String)"> |
| <h3>addArchive</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">addArchive</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path)</span></div> |
| <div class="block">:: Experimental :: |
| Add an archive to be downloaded and unpacked with this Spark job on every node. |
| <p> |
| If an archive is added during execution, it will not be available until the next TaskSet |
| starts. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - can be either a local file, a file in HDFS (or other Hadoop-supported |
| filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, |
| use <code>SparkFiles.get(paths-to-files)</code> to find its download/unpacked location. |
| The given path should be one of .zip, .tar, .tar.gz, .tgz and .jar. |
| <p></dd> |
| <dt>Since:</dt> |
| <dd>3.1.0</dd> |
| <dt>Note:</dt> |
| <dd>A path can be added only once. Subsequent additions of the same path are ignored. |
| <p></dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="listArchives()"> |
| <h3>listArchives</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">listArchives</span>()</div> |
| <div class="block">:: Experimental :: |
| Returns a list of archive paths that are added to resources. |
| <p></div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| <dt>Since:</dt> |
| <dd>3.1.0</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="addFile(java.lang.String,boolean)"> |
| <h3>addFile</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">addFile</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path, |
| boolean recursive)</span></div> |
| <div class="block">Add a file to be downloaded with this Spark job on every node. |
| <p> |
| If a file is added during execution, it will not be available until the next TaskSet starts. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - can be either a local file, a file in HDFS (or other Hadoop-supported |
| filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, |
| use <code>SparkFiles.get(fileName)</code> to find its download location.</dd> |
| <dd><code>recursive</code> - if true, a directory can be given in <code>path</code>. Currently directories are |
| only supported for Hadoop-supported filesystems. |
| <p></dd> |
| <dt>Note:</dt> |
| <dd>A path can be added only once. Subsequent additions of the same path are ignored.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="addSparkListener(org.apache.spark.scheduler.SparkListenerInterface)"> |
| <h3>addSparkListener</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">addSparkListener</span><wbr><span class="parameters">(<a href="scheduler/SparkListenerInterface.html" title="interface in org.apache.spark.scheduler">SparkListenerInterface</a> listener)</span></div> |
| <div class="block">:: DeveloperApi :: |
| Register a listener to receive up-calls from events that happen during execution.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>listener</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="removeSparkListener(org.apache.spark.scheduler.SparkListenerInterface)"> |
| <h3>removeSparkListener</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">removeSparkListener</span><wbr><span class="parameters">(<a href="scheduler/SparkListenerInterface.html" title="interface in org.apache.spark.scheduler">SparkListenerInterface</a> listener)</span></div> |
| <div class="block">:: DeveloperApi :: |
| Deregister the listener from Spark's listener bus.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>listener</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="requestTotalExecutors(int,int,scala.collection.immutable.Map)"> |
| <h3>requestTotalExecutors</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">boolean</span> <span class="element-name">requestTotalExecutors</span><wbr><span class="parameters">(int numExecutors, |
| int localityAwareTasks, |
| scala.collection.immutable.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> hostToLocalTaskCount)</span></div> |
| <div class="block">Update the cluster manager on our scheduling needs. Three bits of information are included |
| to help it make decisions. This applies to the default ResourceProfile.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>numExecutors</code> - The total number of executors we'd like to have. The cluster manager |
| shouldn't kill any running executor to reach this number, but, |
| if all existing executors were to die, this is the number of executors |
| we'd want to be allocated.</dd> |
| <dd><code>localityAwareTasks</code> - The number of tasks in all active stages that have a locality |
| preferences. This includes running, pending, and completed tasks.</dd> |
| <dd><code>hostToLocalTaskCount</code> - A map of hosts to the number of tasks from all active stages |
| that would like to like to run on that host. |
| This includes running, pending, and completed tasks.</dd> |
| <dt>Returns:</dt> |
| <dd>whether the request is acknowledged by the cluster manager.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="requestExecutors(int)"> |
| <h3>requestExecutors</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">boolean</span> <span class="element-name">requestExecutors</span><wbr><span class="parameters">(int numAdditionalExecutors)</span></div> |
| <div class="block">:: DeveloperApi :: |
| Request an additional number of executors from the cluster manager.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>numAdditionalExecutors</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>whether the request is received.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="killExecutors(scala.collection.immutable.Seq)"> |
| <h3>killExecutors</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">boolean</span> <span class="element-name">killExecutors</span><wbr><span class="parameters">(scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>> executorIds)</span></div> |
| <div class="block">:: DeveloperApi :: |
| Request that the cluster manager kill the specified executors. |
| <p> |
| This is not supported when dynamic allocation is turned on. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>executorIds</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>whether the request is received.</dd> |
| <dt>Note:</dt> |
| <dd>This is an indication to the cluster manager that the application wishes to adjust |
| its resource usage downwards. If the application wishes to replace the executors it kills |
| through this method with new ones, it should follow up explicitly with a call to |
| {{SparkContext#requestExecutors}}. |
| <p></dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="killExecutor(java.lang.String)"> |
| <h3>killExecutor</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">boolean</span> <span class="element-name">killExecutor</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> executorId)</span></div> |
| <div class="block">:: DeveloperApi :: |
| Request that the cluster manager kill the specified executor. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>executorId</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>whether the request is received.</dd> |
| <dt>Note:</dt> |
| <dd>This is an indication to the cluster manager that the application wishes to adjust |
| its resource usage downwards. If the application wishes to replace the executor it kills |
| through this method with a new one, it should follow up explicitly with a call to |
| {{SparkContext#requestExecutors}}. |
| <p></dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="version()"> |
| <h3>version</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a></span> <span class="element-name">version</span>()</div> |
| <div class="block">The version of Spark on which this application is running.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getExecutorMemoryStatus()"> |
| <h3>getExecutorMemoryStatus</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>,<wbr>scala.Tuple2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>>></span> <span class="element-name">getExecutorMemoryStatus</span>()</div> |
| <div class="block">Return a map from the block manager to the max memory available for caching and the remaining |
| memory available for caching.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getRDDStorageInfo()"> |
| <h3>getRDDStorageInfo</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type"><a href="storage/RDDInfo.html" title="class in org.apache.spark.storage">RDDInfo</a>[]</span> <span class="element-name">getRDDStorageInfo</span>()</div> |
| <div class="block">:: DeveloperApi :: |
| Return information about what RDDs are cached, if they are in mem or on disk, how much space |
| they take, etc.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getPersistentRDDs()"> |
| <h3>getPersistentRDDs</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.Map<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr><a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><?>></span> <span class="element-name">getPersistentRDDs</span>()</div> |
| <div class="block">Returns an immutable map of RDDs that have marked themselves as persistent via cache() call. |
| <p></div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| <dt>Note:</dt> |
| <dd>This does not necessarily mean the caching or computation was successful.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getAllPools()"> |
| <h3>getAllPools</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Seq<<a href="scheduler/Schedulable.html" title="interface in org.apache.spark.scheduler">Schedulable</a>></span> <span class="element-name">getAllPools</span>()</div> |
| <div class="block">:: DeveloperApi :: |
| Return pools for fair scheduler</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getPoolForName(java.lang.String)"> |
| <h3>getPoolForName</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.Option<<a href="scheduler/Schedulable.html" title="interface in org.apache.spark.scheduler">Schedulable</a>></span> <span class="element-name">getPoolForName</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> pool)</span></div> |
| <div class="block">:: DeveloperApi :: |
| Return the pool associated with the given name, if one exists</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>pool</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getSchedulingMode()"> |
| <h3>getSchedulingMode</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.Enumeration.Value</span> <span class="element-name">getSchedulingMode</span>()</div> |
| <div class="block">Return current scheduling mode</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="addJar(java.lang.String)"> |
| <h3>addJar</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">addJar</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> path)</span></div> |
| <div class="block">Adds a JAR dependency for all tasks to be executed on this <code>SparkContext</code> in the future. |
| <p> |
| If a jar is added during execution, it will not be available until the next TaskSet starts. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>path</code> - can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), |
| an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node. |
| <p></dd> |
| <dt>Note:</dt> |
| <dd>A path can be added only once. Subsequent additions of the same path are ignored.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="listJars()"> |
| <h3>listJars</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">listJars</span>()</div> |
| <div class="block">Returns a list of jar files that are added to resources.</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="stop()"> |
| <h3>stop</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">stop</span>()</div> |
| <div class="block">Shut down the SparkContext.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="stop(int)"> |
| <h3>stop</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">stop</span><wbr><span class="parameters">(int exitCode)</span></div> |
| <div class="block">Shut down the SparkContext with exit code that will passed to scheduler backend. |
| In client mode, client side may call <code>SparkContext.stop()</code> to clean up but exit with |
| code not equal to 0. This behavior cause resource scheduler such as <code>ApplicationMaster</code> |
| exit with success status but client side exited with failed status. Spark can call |
| this method to stop SparkContext and pass client side correct exit code to scheduler backend. |
| Then scheduler backend should send the exit code to corresponding resource scheduler |
| to keep consistent. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>exitCode</code> - Specified exit code that will passed to scheduler backend in client mode.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="setCallSite(java.lang.String)"> |
| <h3>setCallSite</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">setCallSite</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> shortCallSite)</span></div> |
| <div class="block">Set the thread-local property for overriding the call sites |
| of actions and RDDs.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>shortCallSite</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="clearCallSite()"> |
| <h3>clearCallSite</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">clearCallSite</span>()</div> |
| <div class="block">Clear the thread-local property for overriding the call sites |
| of actions and RDDs.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.collection.immutable.Seq,scala.Function2,scala.reflect.ClassTag)"> |
| <h3>runJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U></span> <span class="return-type">void</span> <span class="element-name">runJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.reflect.ClassTag<U> evidence$11)</span></div> |
| <div class="block">Run a function on a given set of partitions in an RDD and pass the results to the given |
| handler function. This is the main entry point for all actions in Spark. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>func</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>partitions</code> - set of partitions to run on; some jobs may not want to compute on all |
| partitions of the target RDD, e.g. for operations like <code>first()</code></dd> |
| <dd><code>resultHandler</code> - callback to pass each result to</dd> |
| <dd><code>evidence$11</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.collection.immutable.Seq,scala.reflect.ClassTag)"> |
| <h3>runJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U></span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></span> <span class="element-name">runJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.reflect.ClassTag<U> evidence$12)</span></div> |
| <div class="block">Run a function on a given set of partitions in an RDD and return the results as an array. |
| The function that is run against each partition additionally takes <code>TaskContext</code> argument. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>func</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>partitions</code> - set of partitions to run on; some jobs may not want to compute on all |
| partitions of the target RDD, e.g. for operations like <code>first()</code></dd> |
| <dd><code>evidence$12</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>in-memory collection with a result of the job (each collection element will contain |
| a result from one partition)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runJob(org.apache.spark.rdd.RDD,scala.Function1,scala.collection.immutable.Seq,scala.reflect.ClassTag)"> |
| <h3>runJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U></span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></span> <span class="element-name">runJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> func, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.reflect.ClassTag<U> evidence$13)</span></div> |
| <div class="block">Run a function on a given set of partitions in an RDD and return the results as an array. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>func</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>partitions</code> - set of partitions to run on; some jobs may not want to compute on all |
| partitions of the target RDD, e.g. for operations like <code>first()</code></dd> |
| <dd><code>evidence$13</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>in-memory collection with a result of the job (each collection element will contain |
| a result from one partition)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.reflect.ClassTag)"> |
| <h3>runJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U></span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></span> <span class="element-name">runJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| scala.reflect.ClassTag<U> evidence$14)</span></div> |
| <div class="block">Run a job on all partitions in an RDD and return the results in an array. The function |
| that is run against each partition additionally takes <code>TaskContext</code> argument. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>func</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>evidence$14</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>in-memory collection with a result of the job (each collection element will contain |
| a result from one partition)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runJob(org.apache.spark.rdd.RDD,scala.Function1,scala.reflect.ClassTag)"> |
| <h3>runJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U></span> <span class="return-type"><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a></span> <span class="element-name">runJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> func, |
| scala.reflect.ClassTag<U> evidence$15)</span></div> |
| <div class="block">Run a job on all partitions in an RDD and return the results in an array. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>func</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>evidence$15</code> - (undocumented)</dd> |
| <dt>Returns:</dt> |
| <dd>in-memory collection with a result of the job (each collection element will contain |
| a result from one partition)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runJob(org.apache.spark.rdd.RDD,scala.Function2,scala.Function2,scala.reflect.ClassTag)"> |
| <h3>runJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U></span> <span class="return-type">void</span> <span class="element-name">runJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> processPartition, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.reflect.ClassTag<U> evidence$16)</span></div> |
| <div class="block">Run a job on all partitions in an RDD and pass the results to a handler function. The function |
| that is run against each partition additionally takes <code>TaskContext</code> argument. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>processPartition</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>resultHandler</code> - callback to pass each result to</dd> |
| <dd><code>evidence$16</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runJob(org.apache.spark.rdd.RDD,scala.Function1,scala.Function2,scala.reflect.ClassTag)"> |
| <h3>runJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U></span> <span class="return-type">void</span> <span class="element-name">runJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> processPartition, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.reflect.ClassTag<U> evidence$17)</span></div> |
| <div class="block">Run a job on all partitions in an RDD and pass the results to a handler function. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>processPartition</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>resultHandler</code> - callback to pass each result to</dd> |
| <dd><code>evidence$17</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="runApproximateJob(org.apache.spark.rdd.RDD,scala.Function2,org.apache.spark.partial.ApproximateEvaluator,long)"> |
| <h3>runApproximateJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U,<wbr> |
| R></span> <span class="return-type"><a href="partial/PartialResult.html" title="class in org.apache.spark.partial">PartialResult</a><R></span> <span class="element-name">runApproximateJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function2<<a href="TaskContext.html" title="class in org.apache.spark">TaskContext</a>,<wbr>scala.collection.Iterator<T>,<wbr>U> func, |
| <a href="partial/ApproximateEvaluator.html" title="interface in org.apache.spark.partial">ApproximateEvaluator</a><U,<wbr>R> evaluator, |
| long timeout)</span></div> |
| <div class="block">:: DeveloperApi :: |
| Run a job that can return approximate results. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>func</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>evaluator</code> - <code>ApproximateEvaluator</code> to receive the partial results</dd> |
| <dd><code>timeout</code> - maximum time to wait for the job, in milliseconds</dd> |
| <dt>Returns:</dt> |
| <dd>partial result (how partial depends on whether the job was finished before or |
| after timeout)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="submitJob(org.apache.spark.rdd.RDD,scala.Function1,scala.collection.immutable.Seq,scala.Function2,scala.Function0)"> |
| <h3>submitJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="type-parameters"><T,<wbr> |
| U,<wbr> |
| R></span> <span class="return-type"><a href="SimpleFutureAction.html" title="class in org.apache.spark">SimpleFutureAction</a><R></span> <span class="element-name">submitJob</span><wbr><span class="parameters">(<a href="rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><T> rdd, |
| scala.Function1<scala.collection.Iterator<T>,<wbr>U> processPartition, |
| scala.collection.immutable.Seq<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>> partitions, |
| scala.Function2<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html" title="class or interface in java.lang" class="external-link">Object</a>,<wbr>U,<wbr>scala.runtime.BoxedUnit> resultHandler, |
| scala.Function0<R> resultFunc)</span></div> |
| <div class="block">Submit a job for execution and return a FutureJob holding the result. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>rdd</code> - target RDD to run tasks on</dd> |
| <dd><code>processPartition</code> - a function to run on each partition of the RDD</dd> |
| <dd><code>partitions</code> - set of partitions to run on; some jobs may not want to compute on all |
| partitions of the target RDD, e.g. for operations like <code>first()</code></dd> |
| <dd><code>resultHandler</code> - callback to pass each result to</dd> |
| <dd><code>resultFunc</code> - function to be executed when the result is ready</dd> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelJobGroup(java.lang.String)"> |
| <h3>cancelJobGroup</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelJobGroup</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> groupId)</span></div> |
| <div class="block">Cancel active jobs for the specified group. See <code>org.apache.spark.SparkContext.setJobGroup</code> |
| for more information.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>groupId</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelJobGroupAndFutureJobs(java.lang.String)"> |
| <h3>cancelJobGroupAndFutureJobs</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelJobGroupAndFutureJobs</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> groupId)</span></div> |
| <div class="block">Cancel active jobs for the specified group, as well as the future jobs in this job group. |
| Note: the maximum number of job groups that can be tracked is set by |
| 'spark.scheduler.numCancelledJobGroupsToTrack'. Once the limit is reached and a new job group |
| is to be added, the oldest job group tracked will be discarded.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>groupId</code> - (undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelJobsWithTag(java.lang.String)"> |
| <h3>cancelJobsWithTag</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelJobsWithTag</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> tag)</span></div> |
| <div class="block">Cancel active jobs that have the specified tag. See <code>org.apache.spark.SparkContext.addJobTag</code>. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>tag</code> - The tag to be cancelled. Cannot contain ',' (comma) character. |
| <p></dd> |
| <dt>Since:</dt> |
| <dd>3.5.0</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelAllJobs()"> |
| <h3>cancelAllJobs</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelAllJobs</span>()</div> |
| <div class="block">Cancel all jobs that have been scheduled or are running.</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelJob(int,java.lang.String)"> |
| <h3>cancelJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelJob</span><wbr><span class="parameters">(int jobId, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> reason)</span></div> |
| <div class="block">Cancel a given job if it's scheduled or running. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>jobId</code> - the job ID to cancel</dd> |
| <dd><code>reason</code> - optional reason for cancellation</dd> |
| <dt>Note:</dt> |
| <dd>Throws <code>InterruptedException</code> if the cancel message cannot be sent</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelJob(int)"> |
| <h3>cancelJob</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelJob</span><wbr><span class="parameters">(int jobId)</span></div> |
| <div class="block">Cancel a given job if it's scheduled or running. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>jobId</code> - the job ID to cancel</dd> |
| <dt>Note:</dt> |
| <dd>Throws <code>InterruptedException</code> if the cancel message cannot be sent</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelStage(int,java.lang.String)"> |
| <h3>cancelStage</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelStage</span><wbr><span class="parameters">(int stageId, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> reason)</span></div> |
| <div class="block">Cancel a given stage and all jobs associated with it. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>stageId</code> - the stage ID to cancel</dd> |
| <dd><code>reason</code> - reason for cancellation</dd> |
| <dt>Note:</dt> |
| <dd>Throws <code>InterruptedException</code> if the cancel message cannot be sent</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="cancelStage(int)"> |
| <h3>cancelStage</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">cancelStage</span><wbr><span class="parameters">(int stageId)</span></div> |
| <div class="block">Cancel a given stage and all jobs associated with it. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>stageId</code> - the stage ID to cancel</dd> |
| <dt>Note:</dt> |
| <dd>Throws <code>InterruptedException</code> if the cancel message cannot be sent</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="killTaskAttempt(long,boolean,java.lang.String)"> |
| <h3>killTaskAttempt</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">boolean</span> <span class="element-name">killTaskAttempt</span><wbr><span class="parameters">(long taskId, |
| boolean interruptThread, |
| <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> reason)</span></div> |
| <div class="block">Kill and reschedule the given task attempt. Task ids can be obtained from the Spark UI |
| or through SparkListener.onTaskStart. |
| <p></div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>taskId</code> - the task ID to kill. This id uniquely identifies the task attempt.</dd> |
| <dd><code>interruptThread</code> - whether to interrupt the thread running the task.</dd> |
| <dd><code>reason</code> - the reason for killing the task, which should be a short string. If a task |
| is killed multiple times with different reasons, only one reason will be reported. |
| <p></dd> |
| <dt>Returns:</dt> |
| <dd>Whether the task was successfully killed.</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="setCheckpointDir(java.lang.String)"> |
| <h3>setCheckpointDir</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">void</span> <span class="element-name">setCheckpointDir</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a> directory)</span></div> |
| <div class="block">Set the directory under which RDDs are going to be checkpointed.</div> |
| <dl class="notes"> |
| <dt>Parameters:</dt> |
| <dd><code>directory</code> - path to the directory where checkpoint files will be stored |
| (must be HDFS path if running in cluster)</dd> |
| </dl> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="getCheckpointDir()"> |
| <h3>getCheckpointDir</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">scala.Option<<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>></span> <span class="element-name">getCheckpointDir</span>()</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="defaultParallelism()"> |
| <h3>defaultParallelism</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">int</span> <span class="element-name">defaultParallelism</span>()</div> |
| <div class="block">Default level of parallelism to use when not given by user (e.g. parallelize and makeRDD).</div> |
| </section> |
| </li> |
| <li> |
| <section class="detail" id="defaultMinPartitions()"> |
| <h3>defaultMinPartitions</h3> |
| <div class="member-signature"><span class="modifiers">public</span> <span class="return-type">int</span> <span class="element-name">defaultMinPartitions</span>()</div> |
| <div class="block">Default min number of partitions for Hadoop RDDs when not given by user |
| Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2. |
| The reasons for this are discussed in https://github.com/mesos/spark/pull/718</div> |
| <dl class="notes"> |
| <dt>Returns:</dt> |
| <dd>(undocumented)</dd> |
| </dl> |
| </section> |
| </li> |
| </ul> |
| </section> |
| </li> |
| </ul> |
| </section> |
| <!-- ========= END OF CLASS DATA ========= --> |
| </main> |
| </div> |
| </div> |
| <script defer="defer" type="text/javascript" src="../../../lib/jquery.js"></script><script defer="defer" type="text/javascript" src="../../../lib/api-javadocs.js"></script></body> |
| </html> |
