<?xml version="1.0"?>
<!--
  ~ Licensed to the Apache Software Foundation (ASF) under one or more
  ~ contributor license agreements.  See the NOTICE file distributed with
  ~ this work for additional information regarding copyright ownership.
  ~ The ASF licenses this file to you under the Apache License, Version 2.0
  ~ (the "License"); you may not use this file except in compliance with
  ~ the License.  You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<!DOCTYPE document[
<!ENTITY sect-num '13'>
<!ENTITY hellip   "&#x02026;" >
]>

<document prev="listeners.html" next="generating-dashboard.html" id="$Id$">

<properties>
  <title>User's Manual: Remote (Distributed) Testing</title>
</properties>

<body>

<section name="&sect-num;. Remote Testing">

<p>In the event that your JMeter client machine is unable, performance-wise, to simulate
enough users to stress your server or is limited at network level, an option exists to control multiple, remote JMeter
engines from a single JMeter client.  By running JMeter remotely, you can replicate
a test across many low-end computers and thus simulate a larger load on the server.  One
instance of the JMeter client can control any number of remote JMeter instances, and collect
all the data from them.  This offers the following features:

<ul>
<li>Saving of test samples to the local machine </li>
<li>Management of multiple JMeterEngines from a single machine </li>
<li>No need to copy the test plan to each server - the client sends it to all the servers</li>
</ul>
</p>
<note>
Note: The same test plan is run by all the servers.
JMeter does not distribute the load between servers, each runs the full test plan.
So if you set 1000 Threads and have 6 JMeter server, you end up injecting 6000 Threads.
</note>
<p>
However, remote mode does use more resources than running the same number of CLI mode tests independently.
If many server instances are used, the client JMeter can become overloaded, as can the client network connection.
This has been improved by switching to Stripped modes (see below) but you should always check that your client is not overloaded.
</p>
<p>Note that while you can execute the JMeterEngine on your application
server, you need to be mindful of the fact that this will be adding processing
overhead on the application server and thus your testing results will be
somewhat tainted.  The recommended approach is to have one or more machines on
the same Ethernet segment as your application server that you configure to run
the JMeter Engine.  This will minimize the impact of the network on the test
results without impacting the performance of the application server
itself.
</p>

<p><b>Step 0: Configure the nodes</b></p>
<p>
Make sure that all the nodes (client and servers) :
<ul>
    <li>are running exactly the same version of JMeter.</li>
    <li>are using the same version of Java on all systems. Using different versions of Java may work but is discouraged.</li>
    <li>have a <a href="#setup_ssl">valid keystore for RMI over SSL</a>, or you have disabled the use of SSL.</li>
</ul>
</p>
<p>
If the test uses any data files, <b>note that these are not sent across by the client so
make sure that these are available in the appropriate directory on each server</b>.
If necessary you can define different values for properties by editing the <code>user.properties</code> or <code>system.properties</code>
files on each server. These properties will be picked up when the server is started and may be
used in the test plan to affect its behaviour (e.g. connecting to a different remote server).
Alternatively use different content in any datafiles used by the test
(e.g. if each server must use unique ids, divide these between the data files)
</p>

<p><b>Step 1: Start the servers </b></p>
<p>To run JMeter in remote node, start the JMeter server component on all machines you wish to run on by running
the <code>JMETER_HOME/bin/jmeter-server</code> (unix) or <code>JMETER_HOME/bin/jmeter-server.bat</code> (windows) script.</p>
<p>Note that there can only be one JMeter server on each node unless different RMI ports are used.</p>
<p>The JMeter server application starts the RMI registry itself; there is no need to start RMI registry separately.</p>
<p>
By default, RMI uses a dynamic port for the JMeter server engine. This can cause problems for firewalls,
so you can define the JMeter property <code>server.rmi.localport</code> to control this port number.
it will be used as the local port number for the server engine.
</p>
<p><b>Step 2: Add the server IP to your client's Properties File</b></p>
<p>Edit the properties file <i>on the controlling JMeter machine</i>.  In <code>JMETER_HOME/bin/jmeter.properties</code>,
find the property named "<code>remote_hosts</code>" and
add the value of your running JMeter server's IP address.  Multiple such servers can be added, comma-delimited.</p>
<p>Note that you can use the <code>-R</code> <a href="get-started.html#override">command line option</a>
instead to specify the remote host(s) to use. This has the same effect as using <code>-r</code> and <code>-Jremote_hosts={serverlist}</code>.
    E.g. </p><source>jmeter -Rhost1,127.0.0.1,host2</source>
<p>If you define the JMeter property <code>server.exitaftertest=true</code>, then the server will exit after it runs a single test.
See also the <code>-X</code> flag (described below)
</p>
<p><b>Step 3a: Start the JMeter Client from a GUI client to check configuration</b></p>
<p>Now you are ready to start the controlling JMeter client. For MS-Windows, start the client with the script "<code>bin/jmeter.bat</code>".  For UNIX,
use the script "<code>bin/jmeter</code>".  You will notice that the Run menu contains two new sub-menus: "Remote Start" and "Remote Stop"
(see figure 1). These menus contain the client that you set in the properties file.  Use the remote start and stop instead of the
normal JMeter start and stop menu items.</p>
<figure image="remote/run-menu00.png" width="487" height="295">Figure 1 - Run Menu</figure>

<p><b>Step 3b: Start the JMeter from a CLI mode Client</b></p>
<p>
GUI mode should only be used for debugging, as a better alternative, you should start the test on remote server(s) from a CLI mode (command-line) client.
The command to do this is:</p>
<source>
jmeter -n -t script.jmx -r
</source>
<p>or</p>
<source>
jmeter -n -t script.jmx -R server1,server2,&hellip;
</source>
<p>
Other flags that may be useful:
</p>
<dl>
<dt><code>-Gproperty=value</code></dt><dd>define a property in all the servers (may appear more than once)</dd>
<dt><code>-X</code></dt><dd>Exit remote servers at the end of the test.</dd>
</dl>
<p>
The first example will start the test on whatever servers are defined in the JMeter property <code>remote_hosts</code>;<br/>
The second example will define <code>remote_hosts</code> from the list of servers and then start the test on the remote servers.
<br/>
The command-line client will exit when all the remote servers have stopped.
</p>
<subsection name="&sect-num;.1 Setting up SSL" anchor="setup_ssl">
<note>Since JMeter 4.0 the default transport mechanism for RMI will use SSL. SSL needs keys and certificates to work. You will have to create those keys yourself.</note>
<p>
The simplest setup is to use one key/cert pair for all JMeter servers and clients you want to connect. JMeter comes with a script to generate a keystore that
contains one key (and its corresponding certificate) named <code>rmi</code>. The script is located in the <code>bin</code> directory and is available for Windows systems (called <code>bin/create-rmi-keystore.bat</code>) and Unix like systems (called <code>bin/create-rmi-keystore.sh</code>). It will generate a key-pair, that is valid for seven days, with a default passphrase of value '<code>changeit</code>'. It is advised to call it from inside the <code>bin</code> directory.
</p>
<p>
When you run the script, it will ask you some questions about some names it will embed in the certificate. You can type in whatever you want, as long the keystore tool accepts it. That value has to match the property <code>server.rmi.ssl.keystore.alias</code>, which defaults to <code>rmi</code>. A sample session to create the keystore is shown below.
</p>
<source>$ cd jmeter/bin
$ ./create-rmi-keystore.sh
What is your first and last name?
  [Unknown]:  rmi
What is the name of your organizational unit?
  [Unknown]:  My unit name
What is the name of your organization?
  [Unknown]:  My organisation name
What is the name of your City or Locality?
  [Unknown]:  Your City
What is the name of your State or Province?
  [Unknown]:  Your State
What is the two-letter country code for this unit?
  [Unknown]:  XY
Is CN=rmi, OU=My unit name, O=My organisation name, L=Your City, ST=Your State, C=XY correct?
  [no]:  yes

Copy the generated rmi_keystore.jks to jmeter/bin folder or reference it in property 'server.rmi.ssl.keystore.file'
</source>
<p>
The <a href="properties_reference.html#remote">defaults settings for RMI</a> should work with this setup. Copy the file <code>bin/rmi_keystore.jks</code> to every JMeter server and client you want to use for your distributed testing setup.
</p>
</subsection>
<subsection name="&sect-num;.2 Doing it Manually" anchor="detail_instructions">
<p>In some cases, the jmeter-server script may not work for you (if you are using an OS platform not anticipated by the JMeter developers).
Here is how to start the JMeter servers (step 1 above) with a more manual process:</p>
<p><b>Step 1a: Start the RMI Registry</b></p>
<p>
Since JMeter 2.3.1, the RMI registry is started by the JMeter server, so this section does not apply in the normal case.
To revert to the previous behaviour, define the JMeter property <code>server.rmi.create=false</code> on the server host systems
and follow the instructions below.
</p>
<p>JMeter uses Remote Method Invocation (RMI) as the remote communication mechanism.  Therefore, you need
to run the RMI Registry application (which is named "<code>rmiregistry</code>") that comes with the JDK and is located in the "<code>bin</code>"
directory.  Before running <code>rmiregistry</code>, make sure that the following jars are in your system classpath:
<ul>
    <li><code>JMETER_HOME/lib/ext/ApacheJMeter_core.jar</code></li>
    <li><code>JMETER_HOME/lib/jorphan.jar</code></li>
    <li><code>JMETER_HOME/lib/logkit-2.0.jar</code></li>
</ul>
The
rmiregistry application needs access to certain JMeter classes.  Run <code>rmiregistry</code> with no parameters.  By default the
application listens to port <code>1099</code>.</p>

<p><b>Step 1b: Start the JMeter Server</b></p>
<p>Once the RMI Registry application is running, start the JMeter Server.
Use the "<code>-s</code>" option with the jmeter startup script ("<code>jmeter -s</code>").</p>

<p>Steps 2 and 3 remain the same.</p>
</subsection>
<subsection name="&sect-num;.3 Tips" anchor="tips">
<p>
JMeter/RMI requires a connection from the client to the server. This will use the port you chose, default <code>1099</code>.<br/>
JMeter/RMI also requires a reverse connection in order to return sample results from the server to the client.<br/>
These will use high-numbered ports. <br/>
These ports can be controlled by jmeter property called <code>client.rmi.localport</code> in <code>jmeter.properties</code>.<br/>
If this is non-zero, it will be used as the base for local port numbers for the client engine. At the moment JMeter will open
up to three ports beginning with the port defined in <code>client.rmi.localport</code>.
If there are any firewalls or other network filters between JMeter client and server,
you will need to make sure that they are set up to allow the connections through.
If necessary, use monitoring software to show what traffic is being generated.
</p>
<p>If you're running Suse Linux, these tips may help. The default installation may enable the firewall. In that case,
remote testing will not work properly. The following tips were contributed by Sergey Ten.</p>
<p>If you see connections refused, turn on debugging by passing the following options.</p>
<source>
rmiregistry -J-Dsun.rmi.log.debug=true \
     -J-Dsun.rmi.server.exceptionTrace=true \
     -J-Dsun.rmi.loader.logLevel=verbose \
     -J-Dsun.rmi.dgc.logLevel=verbose \
     -J-Dsun.rmi.transport.logLevel=verbose \
     -J-Dsun.rmi.transport.tcp.logLevel=verbose \
</source>
<p>Since JMeter 2.3.1, the RMI registry is started by the server; however the options can still be passed in from the JMeter command line.
For example: "<code>jmeter -s -Dsun.rmi.loader.logLevel=verbose</code>" (i.e. omit the <code>-J</code> prefixes).
Alternatively the properties can be defined in the <code>system.properties</code> file.
</p>
<p>The solution to the problem is to remove the loopbacks <code>127.0.0.1</code> and <code>127.0.0.2</code> from <code>/etc/hosts</code>.
What happens is <code>jmeter-server</code> can't connect to rmiregistry if <code>127.0.0.2</code> loopback is not available.
Use the following settings to fix the problem.</p>
<p> Replace</p>
<source>`dirname $0`/jmeter  -s "$@"</source>
<p> With</p>
<source>
HOST="-Djava.rmi.server.hostname=[computer_name][computer_domain] \
      -Djava.security.policy=`dirname $0`/[policy_file]" \
`dirname $0`/jmeter $HOST -s "$@"
</source>
<p>Also create a policy file and add <code>[computer_name][computer_domain]</code> line to <code>/etc/hosts</code>.</p>

<p>In order to better support SSH-tunneling of the RMI communication channels used
in remote testing, since JMeter 2.6:</p>
<ul>
<li>a new property "<code>client.rmi.localport</code>" can be set to control the RMI port used by the RemoteSampleListenerImpl</li>
<li>To support tunneling RMI traffic over an SSH tunnel as the remote endpoint using a port on the local machine,
 loopback interface is now allowed to be used if it has been specified directly using the Java System Property
 "<code>java.rmi.server.hostname</code>" parameter.</li>
</ul>
</subsection>
<subsection name="&sect-num;.4 Using a different port" anchor="portchange">
<p>By default, JMeter uses the standard RMI port <code>1099</code>. It is possible to change this. For this to work successfully,
all the following need to agree:</p>
<ul>
<li>On the server, start <code>rmiregistry</code> using the new port number</li>
<li>On the server, start JMeter with the property <code>server_port</code> defined</li>
<li>On the client, update the <code>remote_hosts</code> property to include the new remote <code>host:port</code> settings</li>
</ul>

<p>Since JMeter 2.1.1, the jmeter-server scripts provide support for changing the port.
For example, assume you want to use port <code>1664</code> (perhaps <code>1099</code> is already used).</p>
On Windows (in a DOS box)
<source>
C:\JMETER> SET SERVER_PORT=1664
C:\JMETER> JMETER-SERVER [other options]
</source>
On Unix:
<source>
$ SERVER_PORT=1664 jmeter-server [other options]
</source>
[N.B. use upper case for the environment variable]
<p>
In both cases, the script starts rmiregistry on the specified port,
and then starts JMeter in server mode, having defined the "<code>server_port</code>" property.
</p>
<p>
The chosen port will be logged in the server <code>jmeter.log</code> file (<code>rmiregistry</code> does not create a log file).
</p>
</subsection>

<subsection name="&sect-num;.5 Using a different sample sender" anchor="sendermode">
<p>
Listeners in the test plan send their results back to the client JMeter which writes the results to the specified files
By default, samples are sent back synchronously as they are generated.
This can affect the maximum throughput of the server test; the sample result has to be sent back before the thread can
continue.
There are some JMeter properties that can be set to alter this behaviour.
</p>
<dl>
<dt><code>mode</code></dt><dd>sample sending mode - default is <code>StrippedBatch</code> since 2.9. This should be set on the client node.
    <dl>
    <dt><code>Standard</code></dt><dd>send samples synchronously as soon as they are generated</dd>
    <dt><code>Hold</code></dt><dd>hold samples in an array until the end of a run. This may use a lot of memory on the server and is discouraged.</dd>
    <dt><code>DiskStore</code></dt><dd>store samples in a disk file (under <code>java.io.temp</code>) until the end of a run.
    The serialised data file is deleted on JVM exit.</dd>
    <dt><code>StrippedDiskStore</code></dt><dd>remove responseData from successful samples, and use DiskStore sender to send them.</dd>
    <dt><code>Batch</code></dt><dd>send saved samples when either the count (<code>num_sample_threshold</code>) or time (<code>time_threshold</code>) exceeds a threshold,
    at which point the samples are sent synchronously.
    The thresholds can be configured on the server using the following properties:
        <dl>
        <dt><code>num_sample_threshold</code></dt><dd>number of samples to accumulate, default <code>100</code></dd>
        <dt><code>time_threshold</code></dt><dd>time threshold, default 60000 ms = 60 seconds</dd>
        </dl>

     See also the Asynch mode, described below.</dd>
    <dt><code>Statistical</code></dt><dd>send a summary sample when either the count or time exceeds a threshold.
    The samples are summarised by thread group name and sample label.
    The following fields are accumulated:
      <ul>
      <li><code>elapsed time</code></li>
      <li><code>latency</code></li>
      <li><code>bytes</code></li>
      <li><code>sample count</code></li>
      <li><code>error count</code></li>
      </ul>
    Other fields that vary between samples are lost.
    </dd>
    <dt><code>Stripped</code></dt><dd>remove responseData from successful samples</dd>
    <dt><code>StrippedBatch</code></dt><dd>remove responseData from successful samples, and use Batch sender to send them.</dd>
    <dt><code>Asynch</code></dt><dd>samples are temporarily stored in a local queue. A separate worker thread sends the samples.
        This allows the test thread to continue without waiting for the result to be sent back to the client.
        However, if samples are being created faster than they can be sent, the queue will eventually fill up,
        and the sampler thread will block until some samples can be drained from the queue.
        This mode is useful for smoothing out peaks in sample generation.
        The queue size can be adjusted by setting the JMeter property
        <code>asynch.batch.queue.size</code> (default <code>100</code>) on the server node.
    </dd>
    <dt><code>StrippedAsynch</code></dt><dd>remove responseData from successful samples, and use Async sender to send them.</dd>
    <dt><code>Custom implementation</code></dt><dd>set the mode parameter to your custom sample sender class name.
    This must implement the interface <code>SampleSender</code> and have a constructor which takes a single
    parameter of type <code>RemoteSampleListener</code>.
    </dd>
    </dl>
    </dd>
</dl>
<note><code>Stripped</code> mode family strips <code>responseData</code> so this means that some Elements that rely
on the previous <code>responseData</code> being available will not work.<br/>
This is not really a problem as there is always a more efficient way to implement this feature.
</note>
<p>The following properties apply to the <code>Batch</code> and <code>Statistical</code> modes:</p>
    <dl>
    <dt><code>num_sample_threshold</code></dt><dd>number of samples in a batch (default <code>100</code>)</dd>
    <dt><code>time_threshold</code></dt><dd>number of milliseconds to wait (default 60 seconds)</dd>
    </dl>
</subsection>


<subsection name="&sect-num;.6 Dealing with nodes that failed starting" anchor="retries">
  <p>
    For large-scale tests there is a chance that some part of remote servers will be unavailable or down.
    For example, when you use automation script to allocate many cloud machines and use them as generators,
    some of requested machines might fail booting because of cloud's issues.
    Since JMeter 2.13 there are new properties to control this behaviour.
  </p>
  <p>
    First what you might want is to retry initialization attempts in hope that failed nodes just slightly delayed their boot.
    To enable retries, you should set <code>client.tries</code> property to total number of connection attempts.
    By default it does only one attempt. To control retry delay, set the <code>client.retries_delay</code> property
    to number of milliseconds to sleep between attempts.
  </p>
  <p>
    Finally, you might still want to run the test with those generators that succeeded initialization and skipping failed nodes.
    To enable that, set the <code>client.continue_on_fail=true</code> property.
  </p>
</subsection>

<subsection name="&sect-num;.7 Using a security-manager" anchor="security-manager">
  <p>When running JMeter in a distributed environment you have to be aware, that JMeter is basically a remote execution agent on both the server and client side. This could be used by a malicious party to gain further access, once it has compromised one of the JMeter clients or servers. To mitigate this Java has the concept of a security manager that gets asked by the JVM before potential dangerous actions are executed. Those actions could be resolving host names, creating or reading files or executing commands in the OS.</p>
  <p>The security manager can be enabled by setting the Java system properties <code>java.security.manager</code> and <code>java.security.policy</code>. Be sure to have a look at the <a href="https://docs.oracle.com/javase/tutorial/security/tour2/index.html">Quick Tour of Controlling Applications</a>.</p>
  <p>Using the new mechansism of <code>setenv.sh</code> (or <code>setenv.bat</code> under Windows) you can enable the security manager by adding the following code snippet to <code>${JMETER_HOME}/bin/setenv.sh</code>:</p>
  <source>JVM_ARGS=" \
   -Djava.security.manager \
   -Djava.security.policy=${JMETER_HOME}/bin/java.policy \
   -Djmeter.home=${JMETER_HOME} \
"</source>
  <p>The JVM will now add the policies defined in the file <code>${JMETER_HOME}/bin/java.policy</code> to the possibly globally defined policies. If you want your definition to be the only source for policies, use two equal signs instead of one when setting the property <code>java.security.policy</code>.</p>
  <p>The policies will be dependent upon your use case and it might take a while to find the correct restricted and allowed actions. Java can help you find the needed policies with the property <code>java.security.debug</code>. Set it to <code>access</code> and it will log all permissions, that it gets asked to allow. Simply add the following line to your <code>setenv.sh</code>:</p>
  <source>JVM_ARGS="${JVM_ARGS} -Djava.security.debug=access"</source>
  <p>It might look a bit strange, that we define a Java system property <code>jmeter.home</code> with the value of <code>${JMETER_HOME}</code>. This variable will be used in the example <code>java.policy</code> to limit the file system access and allow it only to read JMeters configuration and libraries and restrict write access to specific locations, only.</p>
  <p>The following policy definition file has been used for a simple remote test. You will probably have to tweak the policies, when you run more complex scenarios. The test plans are somewhere placed inside the users home directory under a directory called <code>jmeter-testplans</code>. The sample <code>java.policy</code> looks like:</p>
  <source>
grant codeBase "file:${jmeter.home}/bin/*" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/jorphan.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/log4j-api-2.11.1.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/log4j-slf4j-impl-2.11.1.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/slf4j-api-1.7.25.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/log4j-core-2.11.1.jar" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/ext/*" {
  permission java.security.AllPermission;
};

grant codeBase "file:${jmeter.home}/lib/httpclient-4.5.6.jar" {
  permission java.net.SocketPermission "*", "connect,resolve";
};

grant codeBase "file:${jmeter.home}/lib/darcula.jar" {
  permission java.lang.RuntimePermission "modifyThreadGroup";
};

grant codeBase "file:${jmeter.home}/lib/xercesImpl-2.12.0.jar" {
  permission java.io.FilePermission "${java.home}/lib/xerces.properties", "read";
};

grant codeBase "file:${jmeter.home}/lib/groovy-all-2.4.15.jar" {
  permission groovy.security.GroovyCodeSourcePermission "/groovy/script";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.reflect";
  permission java.lang.RuntimePermission "getProtectionDomain";
};

grant {
  permission java.io.FilePermission "${jmeter.home}/backups", "read,write";
  permission java.io.FilePermission "${jmeter.home}/backups/*", "read,write,delete";
  permission java.io.FilePermission "${jmeter.home}/bin/upgrade.properties", "read";
  permission java.io.FilePermission "${jmeter.home}/lib/ext/-", "read";
  permission java.io.FilePermission "${jmeter.home}/lib/ext", "read";
  permission java.io.FilePermission "${jmeter.home}/lib/-", "read";
  permission java.io.FilePermission "${user.home}/jmeter-testplans/-", "read,write";
  permission java.io.SerializablePermission "enableSubclassImplementation";
  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
  permission java.lang.RuntimePermission "accessClassInPackage.jdk.internal.dynalink.support";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.awt";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.misc";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.swing";
  permission java.lang.RuntimePermission "accessDeclaredMembers";
  permission java.lang.RuntimePermission "createClassLoader";
  permission java.lang.RuntimePermission "createSecurityManager";
  permission java.lang.RuntimePermission "getClassLoader";
  permission java.lang.RuntimePermission "getenv.*";
  permission java.lang.RuntimePermission "nashorn.createGlobal";
  permission java.util.PropertyPermission "*", "read";
};
  </source>
  <note>The usage of <code>java.security.AllPermission</code> is an easy way to make your test plans work, but it might be a dangerous shortcut on your path to security.</note>
</subsection>

</section>

</body>
</document>