Correct longAsInt output in the stress test application
RNG-171 changed the cached int source to output a long as two ints using
the low 32-bits, then high 32-bits.
Updated the RngDataOutput to support HiLo or LoHi. Added tests for each
implementation.
Dropped high-bits and low-bits command options. Added source64 option
with enum for high, low, high-Low, low-high, long, int. The default for
stress testing matches the caching implementation.
Update the test documentation to reflect the changes.
diff --git a/commons-rng-examples/examples-stress/endianness.md b/commons-rng-examples/examples-stress/endianness.md
index 56567f8..95bc9b9 100644
--- a/commons-rng-examples/examples-stress/endianness.md
+++ b/commons-rng-examples/examples-stress/endianness.md
@@ -59,12 +59,12 @@
contains a simple command to output data from a random generator. To output data in both text
and binary format use the following commands:
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
+ java -jar target/examples-stress.jar output KISS -s 1 -n 2048000 \
-f DIEHARDER -o test.dh
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
- -f BINARY -o test.big -b big_endian
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
- -f BINARY -o test.little -b little_endian
+ java -jar target/examples-stress.jar output KISS -s 1 -n 1000 \
+ -f BINARY -o test.big -b big_endian --buffer-size 8192
+ java -jar target/examples-stress.jar output KISS -s 1 -n 1000 \
+ -f BINARY -o test.little -b little_endian --buffer-size 8192
This should produce the following output files:
@@ -74,6 +74,8 @@
| test.big | Binary file using the big-endian format |
| test.little | Binary file using the little-endian format |
+Note that the `-n` parameter is the count of numbers in text output but the count of buffers in binary output. The example above has 2048 4-byte integers per 8192 buffer. The `-n` parameter has been adjusted so the output numbers are the same.
+
The data can then be used to run a test within **Dieharder**:
dieharder -g 202 -d 0 -f test.dh
@@ -89,10 +91,10 @@
In the following example the stress test application directly writes to `stdout` which is then
piped to the `dieharder` application which reads using `stdin` (`-g 200` option):
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
+ java -jar target/examples-stress.jar output KISS -s 1 -n 20000000 \
-f DIEHARDER -o test.dh
dieharder -g 202 -d 0 -f test.dh
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
+ java -jar target/examples-stress.jar output KISS -s 1 -n 10000 \
-f BINARY -b little_endian | dieharder -g 200 -d 0
If the results are not the same then the second command can be repeated with `-b big_endian`.
@@ -150,7 +152,7 @@
01011100 10001111 11110001 11000001 1552937409 1552937409
10110000 01110101 10010011 00011100 2960495388 -1334471908
-The `stdin2testu01` has been written to output the same format when using the `raw32` mode.
+The `stdin2testu01` has been written to output in the same format when using the `raw32` mode.
If the data has been correctly read the `bridge.data` and `bridge.out` should match.
If the endianess is incorrect then the data sent by the Java application will not match the
data read by the sub-process. For example to swap the endianness use the `-b` option:
@@ -215,7 +217,7 @@
The only difference should be the the header and footer added by the `stress` command
to the results file. If the endianness is incorrect for the `output` command then the number
data will not match. Note that the `output` command by default uses big endian for consistency
-across all platforms.
+across all platforms; the `stress` command uses the native byte order of the platform.
An equivalent 64-bit output would use the `--raw64` option for each command:
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java
index f32e71c..70a2a04 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java
@@ -131,8 +131,8 @@
// Open the stdin of the process and write to a custom data sink.
// Note: The 'bridge' command only supports 32-bit data in order to
// demonstrate passing suitable data for TestU01 BigCrush.
- final boolean raw64 = false;
- try (RngDataOutput sink = RNGUtils.createDataOutput(rng, raw64,
+ final Source64Mode source64 = null;
+ try (RngDataOutput sink = RNGUtils.createDataOutput(rng, source64,
testingProcess.getOutputStream(), buffer.capacity() * 4, byteOrder)) {
sink.write(rng);
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java
index 6b0765a..bb2a7bc 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java
@@ -126,25 +126,22 @@
description = {"Reverse the bits in the data (default: ${DEFAULT-VALUE})."})
private boolean reverseBits;
- /** Flag to use the upper 32-bits from the 64-bit long output. */
- @Option(names = {"--high-bits"},
- description = {"Use the upper 32-bits from the 64-bit long output.",
- "Takes precedent over --low-bits."})
- private boolean longHighBits;
-
- /** Flag to use the lower 32-bits from the 64-bit long output. */
- @Option(names = {"--low-bits"},
- description = {"Use the lower 32-bits from the 64-bit long output."})
- private boolean longLowBits;
-
/** Flag to use 64-bit long output. */
@Option(names = {"--raw64"},
description = {"Use 64-bit output (default is 32-bit).",
"This is ignored if not a native 64-bit generator.",
- "In 32-bit mode the output uses the upper then lower bits of 64-bit " +
- "generators sequentially."})
+ "Set to true sets the source64 mode to LONG."})
private boolean raw64;
+ /** Output mode for 64-bit long output. */
+ @Option(names = {"--source64"},
+ description = {"Output mode for 64-bit generators (default: ${DEFAULT-VALUE}).",
+ "This is ignored if not a native 64-bit generator.",
+ "In 32-bit mode the output uses a combination of upper and " +
+ "lower bits of the 64-bit value.",
+ "Valid values: ${COMPLETION-CANDIDATES}."})
+ private Source64Mode source64 = RNGUtils.getSource64Default();
+
/**
* The output mode for existing files.
*/
@@ -166,12 +163,20 @@
final Object objectSeed = createSeed();
UniformRandomProvider rng = createRNG(objectSeed);
+ // raw64 flag overrides the source64 mode
+ if (raw64) {
+ source64 = Source64Mode.LONG;
+ }
+ if (source64 == Source64Mode.LONG && !(rng instanceof RandomLongSource)) {
+ throw new ApplicationException("Not a 64-bit RNG: " + rng);
+ }
+
// Upper or lower bits from 64-bit generators must be created first.
- // This will throw if not a 64-bit generator.
- if (longHighBits) {
- rng = RNGUtils.createLongUpperBitsIntProvider(rng);
- } else if (longLowBits) {
- rng = RNGUtils.createLongLowerBitsIntProvider(rng);
+ // Note this does not test source64 != Source64Mode.LONG as the full long
+ // output split into hi-lo or lo-hi is supported by the RngDataOutput.
+ if (rng instanceof RandomLongSource &&
+ (source64 == Source64Mode.HI || source64 == Source64Mode.LO || source64 == Source64Mode.INT)) {
+ rng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
if (reverseBits) {
rng = RNGUtils.createReverseBitsProvider(rng);
@@ -305,9 +310,9 @@
*/
private UniformRandomProvider toOutputFormat(UniformRandomProvider rng) {
UniformRandomProvider convertedRng = rng;
- if (rng instanceof RandomLongSource && !raw64) {
+ if (rng instanceof RandomLongSource && source64 != Source64Mode.LONG) {
// Convert to 32-bit generator
- convertedRng = RNGUtils.createIntProvider(rng);
+ convertedRng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
if (byteOrder == ByteOrder.LITTLE_ENDIAN) {
convertedRng = RNGUtils.createReverseBytesProvider(convertedRng);
@@ -421,7 +426,7 @@
// If count is not positive use max value.
// This is effectively unlimited: program must be killed.
final long limit = (count < 1) ? Long.MAX_VALUE : count;
- try (RngDataOutput data = RNGUtils.createDataOutput(rng, raw64, out, bufferSize, byteOrder)) {
+ try (RngDataOutput data = RNGUtils.createDataOutput(rng, source64, out, bufferSize, byteOrder)) {
for (long c = 0; c < limit; c++) {
data.write(rng);
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java
index 29c09a5..4d799c9 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java
@@ -33,7 +33,7 @@
final class RNGUtils {
/** Name prefix for bit-reversed RNGs. */
private static final String BYTE_REVERSED = "Byte-reversed ";
- /** Name prefix for byte-reversed RNGs. */
+ /** Name prefix for bit-reversed RNGs. */
private static final String BIT_REVERSED = "Bit-reversed ";
/** Name prefix for hashcode mixed RNGs. */
private static final String HASH_CODE = "HashCode ^ ";
@@ -41,15 +41,26 @@
private static final String TLR_MIXED = "ThreadLocalRandom ^ ";
/** Name of xor operator for xor mixed RNGs. */
private static final String XOR = " ^ ";
+ /** Message for an unrecognised source64 mode. */
+ private static final String UNRECOGNISED_SOURCE_64_MODE = "Unrecognised source64 mode: ";
/** Message for an unrecognised native output type. */
private static final String UNRECOGNISED_NATIVE_TYPE = "Unrecognised native output type: ";
- /** Message when not a RandomLongSource. */
- private static final String NOT_LONG_SOURCE = "Not a 64-bit long generator: ";
+ /** The source64 mode for the default LongProvider caching implementation. */
+ private static final Source64Mode SOURCE_64_DEFAULT = Source64Mode.LO_HI;
/** No public construction. */
private RNGUtils() {}
/**
+ * Gets the source64 mode for the default caching implementation in {@link LongProvider}.
+ *
+ * @return the source64 default mode
+ */
+ static Source64Mode getSource64Default() {
+ return SOURCE_64_DEFAULT;
+ }
+
+ /**
* Wrap the random generator with a new instance that will reverse the byte order of
* the native type. The input must be either a {@link RandomIntSource} or
* {@link RandomLongSource}.
@@ -132,6 +143,35 @@
}
/**
+ * Wrap the {@link RandomLongSource} with an {@link IntProvider} that will use the
+ * specified part of {@link RandomLongSource#next()} to create the int value.
+ *
+ * @param <R> The type of the generator.
+ * @param rng The random generator.
+ * @param mode the mode
+ * @return the int random generator.
+ * @throws ApplicationException If the input source native type is not 64-bit.
+ */
+ static <R extends RandomLongSource & UniformRandomProvider>
+ UniformRandomProvider createIntProvider(final R rng, Source64Mode mode) {
+ switch (mode) {
+ case INT:
+ return createIntProvider(rng);
+ case LO_HI:
+ return createLongLowerUpperBitsIntProvider(rng);
+ case HI_LO:
+ return createLongUpperLowerBitsIntProvider(rng);
+ case HI:
+ return createLongUpperBitsIntProvider(rng);
+ case LO:
+ return createLongLowerBitsIntProvider(rng);
+ case LONG:
+ default:
+ throw new IllegalArgumentException("Unsupported mode " + mode);
+ }
+ }
+
+ /**
* Wrap the random generator with an {@link IntProvider} that will use
* {@link UniformRandomProvider#nextInt()}.
* An input {@link RandomIntSource} is returned unmodified.
@@ -139,7 +179,7 @@
* @param rng The random generator.
* @return the int random generator.
*/
- static UniformRandomProvider createIntProvider(final UniformRandomProvider rng) {
+ private static UniformRandomProvider createIntProvider(final UniformRandomProvider rng) {
if (!(rng instanceof RandomIntSource)) {
return new IntProvider() {
@Override
@@ -157,6 +197,78 @@
}
/**
+ * Wrap the random generator with an {@link IntProvider} that will use the lower then upper
+ * 32-bits from {@link UniformRandomProvider#nextLong()}.
+ * An input {@link RandomIntSource} is returned unmodified.
+ *
+ * @param rng The random generator.
+ * @return the int random generator.
+ */
+ private static UniformRandomProvider createLongLowerUpperBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ private long source = -1;
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.next();
+ // store hi
+ source = next >>> 32;
+ // extract low
+ return (int) next;
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+
+ @Override
+ public String toString() {
+ return "Long lower-upper bits " + rng.toString();
+ }
+ };
+ }
+
+ /**
+ * Wrap the random generator with an {@link IntProvider} that will use the lower then upper
+ * 32-bits from {@link UniformRandomProvider#nextLong()}.
+ * An input {@link RandomIntSource} is returned unmodified.
+ *
+ * @param rng The random generator.
+ * @return the int random generator.
+ */
+ private static UniformRandomProvider createLongUpperLowerBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ private long source = -1;
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.next();
+ // store low
+ source = next & 0xffff_ffffL;
+ // extract hi
+ return (int) (next >>> 32);
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+
+ @Override
+ public String toString() {
+ return "Long upper-lower bits " + rng.toString();
+ }
+ };
+ }
+
+ /**
* Wrap the random generator with an {@link IntProvider} that will use the upper
* 32-bits of the {@code long} from {@link UniformRandomProvider#nextLong()}.
* The input must be a {@link RandomLongSource}.
@@ -165,21 +277,18 @@
* @return the upper bits random generator.
* @throws ApplicationException If the input source native type is not 64-bit.
*/
- static UniformRandomProvider createLongUpperBitsIntProvider(final UniformRandomProvider rng) {
- if (rng instanceof RandomLongSource) {
- return new IntProvider() {
- @Override
- public int next() {
- return (int) (rng.nextLong() >>> 32);
- }
+ private static UniformRandomProvider createLongUpperBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ @Override
+ public int next() {
+ return (int) (rng.next() >>> 32);
+ }
- @Override
- public String toString() {
- return "Long upper-bits " + rng.toString();
- }
- };
- }
- throw new ApplicationException(NOT_LONG_SOURCE + rng);
+ @Override
+ public String toString() {
+ return "Long upper-bits " + rng.toString();
+ }
+ };
}
/**
@@ -191,21 +300,18 @@
* @return the lower bits random generator.
* @throws ApplicationException If the input source native type is not 64-bit.
*/
- static UniformRandomProvider createLongLowerBitsIntProvider(final UniformRandomProvider rng) {
- if (rng instanceof RandomLongSource) {
- return new IntProvider() {
- @Override
- public int next() {
- return (int) rng.nextLong();
- }
+ private static UniformRandomProvider createLongLowerBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ @Override
+ public int next() {
+ return (int) rng.next();
+ }
- @Override
- public String toString() {
- return "Long lower-bits " + rng.toString();
- }
- };
- }
- throw new ApplicationException(NOT_LONG_SOURCE + rng);
+ @Override
+ public String toString() {
+ return "Long lower-bits " + rng.toString();
+ }
+ };
}
/**
@@ -360,32 +466,60 @@
*
* <p>If the RNG is a {@link RandomLongSource} then the byte output can be 32-bit or 64-bit.
* If 32-bit then the 64-bit output will be written as if 2 {@code int} values were generated
- * sequentially from the upper then lower 32-bits of the {@code long}. This setting is
- * significant depending on the byte order. If using the Java standard big-endian
- * representation the flag has no effect and the output will be the same. If using little
- * endian the output bytes will be written as:</p>
+ * sequentially from the {@code long} (order depends on the source64 mode). This setting is
+ * significant depending on the byte order. For example for a high-low source64 mode and
+ * using the Java standard big-endian representation the output is the same as the raw 64-bit
+ * output. If using little endian the output bytes will be written as:</p>
*
* <pre>
* 76543210 -> 4567 0123
* </pre>
*
+ * <h2>Note</h2>
+ *
+ * <p>The output from an implementation of RandomLongSource from the RNG core package
+ * may output the long bits as integers: in high-low order; in low-high order; using
+ * only the low bits; using only the high bits; or other combinations. This method
+ * allows testing the long output as if it were two int outputs, i.e. using the full
+ * bit output of a long provider with a stress test application that targets 32-bit
+ * random values (e.g. Test U01).
+ *
+ * <p>The results of stress testing can be used to determine if the provider
+ * implementation can use the upper, lower or both parts of the long output for int
+ * generation. In the case of the combined upper-lower output it is not expected that
+ * the order low-high or high-low is important given the stress test will consume
+ * thousands of numbers per test. The default 32-bit mode for a 64-bit source is high-low
+ * for backwards compatibility.
+ *
* @param rng The random generator.
- * @param raw64 Set to true for 64-bit byte output.
+ * @param source64 The output mode for a 64-bit source
* @param out Output stream.
* @param byteSize Number of bytes values to write.
* @param byteOrder Byte order.
* @return the data output
- * @throws ApplicationException If the input source native type is not recognised.
+ * @throws ApplicationException If the input source native type is not recognised; or if
+ * the mode for a RandomLongSource is not one of: raw; hi-lo; or lo-hi.
*/
- static RngDataOutput createDataOutput(final UniformRandomProvider rng, boolean raw64,
+ static RngDataOutput createDataOutput(final UniformRandomProvider rng, Source64Mode source64,
OutputStream out, int byteSize, ByteOrder byteOrder) {
if (rng instanceof RandomIntSource) {
return RngDataOutput.ofInt(out, byteSize / 4, byteOrder);
}
if (rng instanceof RandomLongSource) {
- return raw64 ?
- RngDataOutput.ofLong(out, byteSize / 8, byteOrder) :
- RngDataOutput.ofLongAsInt(out, byteSize / 8, byteOrder);
+ switch (source64) {
+ case HI_LO:
+ return RngDataOutput.ofLongAsHLInt(out, byteSize / 8, byteOrder);
+ case LO_HI:
+ return RngDataOutput.ofLongAsLHInt(out, byteSize / 8, byteOrder);
+ case LONG:
+ return RngDataOutput.ofLong(out, byteSize / 8, byteOrder);
+ // Note other options should have already been converted to an IntProvider
+ case INT:
+ case LO:
+ case HI:
+ default:
+ throw new ApplicationException(UNRECOGNISED_SOURCE_64_MODE + source64);
+ }
}
throw new ApplicationException(UNRECOGNISED_NATIVE_TYPE + rng);
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java
index 90972d6..5abeccf 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java
@@ -54,6 +54,11 @@
* {@link java.io.BufferedOutputStream#write(int) BufferedOutputStream#write(int)} that
* occur for each {@code int} value that is written to
* {@link java.io.DataOutputStream#writeInt(int) DataOutputStream#writeInt(int)}.</p>
+ *
+ * <p>This class has adaptors to write the long output from a RNG to two int values.
+ * To match the caching implementation in the the core LongProvider class this is tested
+ * to output int values in the same order as an instance of the LongProvider. Currently
+ * this outputs in order: low 32-bits, high 32-bits.
*/
abstract class RngDataOutput implements Closeable {
/** The data buffer. */
@@ -155,7 +160,8 @@
}
/**
- * Write {@code long} data as two little-endian {@code int} values.
+ * Write {@code long} data as two little-endian {@code int} values, high 32-bits then
+ * low 32-bits.
* <pre>
* 76543210 -> 4567 0123
* </pre>
@@ -180,7 +186,39 @@
@Override
public void fillBuffer(UniformRandomProvider rng) {
for (int i = 0; i < buffer.length; i += 8) {
- writeLongAsIntLE(i, rng.nextLong());
+ writeLongAsHighLowIntLE(i, rng.nextLong());
+ }
+ }
+ }
+
+ /**
+ * Write {@code long} data as two big-endian {@code int} values, low 32-bits then
+ * high 32-bits.
+ * <pre>
+ * 76543210 -> 3210 7654
+ * </pre>
+ *
+ * <p>This is a specialisation that allows the Java big-endian representation to be split
+ * into two big-endian values in the original order of lower then upper bits. In
+ * comparison the {@link BLongRngDataOutput} will output the same data as:
+ *
+ * <pre>
+ * 76543210 -> 7654 3210
+ * </pre>
+ */
+ private static class BLongAsLoHiIntRngDataOutput extends RngDataOutput {
+ /**
+ * @param out Output stream.
+ * @param size Buffer size.
+ */
+ BLongAsLoHiIntRngDataOutput(OutputStream out, int size) {
+ super(out, size);
+ }
+
+ @Override
+ public void fillBuffer(UniformRandomProvider rng) {
+ for (int i = 0; i < buffer.length; i += 8) {
+ writeLongAsLowHighIntBE(i, rng.nextLong());
}
}
}
@@ -258,7 +296,7 @@
}
/**
- * Writes an {@code long} to the buffer as eight bytes, low byte first (big-endian).
+ * Writes an {@code long} to the buffer as eight bytes, low byte first (little-endian).
*
* @param index the index to start writing.
* @param value an {@code long} to be written.
@@ -276,22 +314,50 @@
/**
* Writes an {@code long} to the buffer as two integers of four bytes, each
- * low byte first (big-endian).
+ * low byte first (little-endian). The long is written as the high 32-bits,
+ * then the low 32-bits.
+ *
+ * <p>Note: A LowHigh little-endian output is the same as {@link #writeLongLE(int, long)}.
*
* @param index the index to start writing.
* @param value an {@code long} to be written.
*/
- final void writeLongAsIntLE(int index, long value) {
+ final void writeLongAsHighLowIntLE(int index, long value) {
+ // high
buffer[index ] = (byte) (value >>> 32);
buffer[index + 1] = (byte) (value >>> 40);
buffer[index + 2] = (byte) (value >>> 48);
buffer[index + 3] = (byte) (value >>> 56);
+ // low
buffer[index + 4] = (byte) value;
buffer[index + 5] = (byte) (value >>> 8);
buffer[index + 6] = (byte) (value >>> 16);
buffer[index + 7] = (byte) (value >>> 24);
}
+ /**
+ * Writes an {@code long} to the buffer as two integers of four bytes, each
+ * high byte first (big-endian). The long is written as the low 32-bits,
+ * then the high 32-bits.
+ *
+ * <p>Note: A HighLow big-endian output is the same as {@link #writeLongBE(int, long)}.
+ *
+ * @param index the index to start writing.
+ * @param value an {@code long} to be written.
+ */
+ final void writeLongAsLowHighIntBE(int index, long value) {
+ // low
+ buffer[index ] = (byte) (value >>> 24);
+ buffer[index + 1] = (byte) (value >>> 16);
+ buffer[index + 2] = (byte) (value >>> 8);
+ buffer[index + 3] = (byte) value;
+ // high
+ buffer[index + 4] = (byte) (value >>> 56);
+ buffer[index + 5] = (byte) (value >>> 48);
+ buffer[index + 6] = (byte) (value >>> 40);
+ buffer[index + 7] = (byte) (value >>> 32);
+ }
+
@Override
public void close() throws IOException {
try (OutputStream ostream = out) {
@@ -338,7 +404,7 @@
/**
* Create a new instance to write batches of data from
* {@link UniformRandomProvider#nextLong()} to the specified output as two sequential
- * {@code int} values.
+ * {@code int} values, high 32-bits then low 32-bits.
*
* <p>This will output the following bytes:</p>
*
@@ -359,11 +425,43 @@
* @return the data output
*/
@SuppressWarnings("resource")
- static RngDataOutput ofLongAsInt(OutputStream out, int size, ByteOrder byteOrder) {
+ static RngDataOutput ofLongAsHLInt(OutputStream out, int size, ByteOrder byteOrder) {
// Ensure the buffer is positive and a factor of 8
final int bytes = Math.max(size * 8, 8);
return byteOrder == ByteOrder.LITTLE_ENDIAN ?
new LLongAsIntRngDataOutput(out, bytes) :
new BLongRngDataOutput(out, bytes);
}
+
+ /**
+ * Create a new instance to write batches of data from
+ * {@link UniformRandomProvider#nextLong()} to the specified output as two sequential
+ * {@code int} values, low 32-bits then high 32-bits.
+ *
+ * <p>This will output the following bytes:</p>
+ *
+ * <pre>
+ * // Little-endian
+ * 76543210 -> 0123 4567
+ *
+ * // Big-endian
+ * 76543210 -> 3210 7654
+ * </pre>
+ *
+ * <p>This ensures the output from the generator is the original lower then upper order bits
+ * for each endianess.
+ *
+ * @param out Output stream.
+ * @param size Number of values to write.
+ * @param byteOrder Byte order.
+ * @return the data output
+ */
+ @SuppressWarnings("resource")
+ static RngDataOutput ofLongAsLHInt(OutputStream out, int size, ByteOrder byteOrder) {
+ // Ensure the buffer is positive and a factor of 8
+ final int bytes = Math.max(size * 8, 8);
+ return byteOrder == ByteOrder.LITTLE_ENDIAN ?
+ new LLongRngDataOutput(out, bytes) :
+ new BLongAsLoHiIntRngDataOutput(out, bytes);
+ }
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/Source64Mode.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/Source64Mode.java
new file mode 100644
index 0000000..357516a
--- /dev/null
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/Source64Mode.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.rng.examples.stress;
+
+/**
+ * The mode to output a 64-bit source of randomness.
+ */
+enum Source64Mode {
+ /** The 64-bit output from nextLong. */
+ LONG,
+ /** The 32-bit output from nextInt. */
+ INT,
+ /** The high 32-bits, then low 32-bits of the 64-bit output. */
+ HI_LO,
+ /** The low 32-bits, then high 32-bits of the 64-bit output. */
+ LO_HI,
+ /** The low 32-bits of the 64-bit output. */
+ LO,
+ /** The high 32-bits of the 64-bit output. */
+ HI;
+}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java
index 62c4782..b24068e 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java
@@ -153,25 +153,39 @@
"when passing using the standard sequence."})
private boolean reverseBits;
- /** Flag to use the upper 32-bits from the 64-bit long output. */
- @Option(names = {"--high-bits"},
- description = {"Use the upper 32-bits from the 64-bit long output.",
- "Takes precedent over --low-bits."})
- private boolean longHighBits;
-
- /** Flag to use the lower 32-bits from the 64-bit long output. */
- @Option(names = {"--low-bits"},
- description = {"Use the lower 32-bits from the 64-bit long output."})
- private boolean longLowBits;
-
/** Flag to use 64-bit long output. */
@Option(names = {"--raw64"},
description = {"Use 64-bit output (default is 32-bit).",
- "This requires a 64-bit testing application and native 64-bit generators.",
- "In 32-bit mode the output uses the upper then lower bits of 64-bit " +
- "generators sequentially, each appropriately byte reversed for the platform."})
+ "This is ignored if not a native 64-bit generator.",
+ "Set to true sets the source64 mode to LONG."})
private boolean raw64;
+ /** Output mode for 64-bit long output.
+ *
+ * <p>Note: The default is set as the default caching implementation.
+ * It passes the full output of the RNG to the stress test application.
+ * Any combination random sources are performed on the full 64-bit output.
+ *
+ * <p>If using INT this will use the RNG's nextInt method.
+ * Any combination random sources are performed on the 32-bit output. Without
+ * a combination random source the output should be the same as the default if
+ * the generator uses the default caching implementation.
+ *
+ * <p>LONG and LO_HI should match binary output when LITTLE_ENDIAN. LONG and HI_LO
+ * should match binary output when BIG_ENDIAN.
+ *
+ * <p>Changing from HI_LO to LO_HI should not effect the stress test as many values are consumed
+ * per test. Using HI or LO may have a different outcome as parts of the generator output
+ * may be weak, e.g. the lower bits of linear congruential generators.
+ */
+ @Option(names = {"--source64"},
+ description = {"Output mode for 64-bit generators (default: ${DEFAULT-VALUE}).",
+ "This is ignored if not a native 64-bit generator.",
+ "In 32-bit mode the output uses a combination of upper and " +
+ "lower bits of the 64-bit value.",
+ "Valid values: ${COMPLETION-CANDIDATES}."})
+ private Source64Mode source64 = RNGUtils.getSource64Default();
+
/** The random seed as a byte[]. */
@Option(names = {"-x", "--hex-seed"},
description = {"The hex-encoded random seed.",
@@ -481,6 +495,11 @@
String basePath,
Iterable<StressTestData> stressTestData,
ProgressTracker progressTracker) {
+ // raw64 flag overrides the source64 mode
+ if (raw64) {
+ source64 = Source64Mode.LONG;
+ }
+
final List<Runnable> tasks = new ArrayList<>();
for (final StressTestData testData : stressTestData) {
for (int trial = 1; trial <= testData.getTrials(); trial++) {
@@ -501,12 +520,17 @@
final byte[] seed = createSeed(testData.getRandomSource());
UniformRandomProvider rng = testData.createRNG(seed);
- // Upper or lower bits from 64-bit generators must be created first.
- // This will throw if not a 64-bit generator.
- if (longHighBits) {
- rng = RNGUtils.createLongUpperBitsIntProvider(rng);
- } else if (longLowBits) {
- rng = RNGUtils.createLongLowerBitsIntProvider(rng);
+ if (source64 == Source64Mode.LONG && !(rng instanceof RandomLongSource)) {
+ throw new ApplicationException("Not a 64-bit RNG: " + rng);
+ }
+
+ // Upper or lower bits from 64-bit generators must be created first before
+ // any further combination operators.
+ // Note this does not test source64 != Source64Mode.LONG as the full long
+ // output split into hi-lo or lo-hi is supported by the RngDataOutput.
+ if (rng instanceof RandomLongSource &&
+ (source64 == Source64Mode.HI || source64 == Source64Mode.LO || source64 == Source64Mode.INT)) {
+ rng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
// Combination generators. Mainly used for testing.
@@ -669,7 +693,7 @@
// of a new one)
// -- There are no pending tasks (i.e. the final submission or the end of a final task)
if (completed >= total ||
- (current >= nextReportTimestamp && (running == parallelTasks || pending == 0))) {
+ (current >= nextReportTimestamp && running == parallelTasks || pending == 0)) {
// Report
nextReportTimestamp = current + PROGRESS_INTERVAL;
final StringBuilder sb = createStringBuilderWithTimestamp(current, pending, running, completed);
@@ -991,7 +1015,7 @@
final Process testingProcess = builder.start();
// Use a custom data output to write the RNG.
- try (RngDataOutput sink = RNGUtils.createDataOutput(rng, cmd.raw64,
+ try (RngDataOutput sink = RNGUtils.createDataOutput(rng, cmd.source64,
testingProcess.getOutputStream(), cmd.bufferSize, cmd.byteOrder)) {
for (;;) {
sink.write(rng);
@@ -1040,7 +1064,7 @@
.append(C).append("Native byte-order: ").append(ByteOrder.nativeOrder()).append(N)
.append(C).append("Output byte-order: ").append(cmd.byteOrder).append(N);
if (rng instanceof RandomLongSource) {
- sb.append(C).append("64-bit output: ").append(cmd.raw64).append(N);
+ sb.append(C).append("64-bit output: ").append(cmd.source64).append(N);
}
sb.append(C).append(N)
.append(C).append("Analyzer: ");
diff --git a/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RNGUtilsTest.java b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RNGUtilsTest.java
new file mode 100644
index 0000000..55084e1
--- /dev/null
+++ b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RNGUtilsTest.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.rng.examples.stress;
+
+import org.apache.commons.rng.UniformRandomProvider;
+import org.apache.commons.rng.core.source64.SplitMix64;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for {@link RNGUtils}.
+ */
+class RNGUtilsTest {
+ @Test
+ void testCreateIntProviderLongThrows() {
+ final SplitMix64 rng = new SplitMix64(42);
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> RNGUtils.createIntProvider(rng, Source64Mode.LONG));
+ }
+
+ @Test
+ void testCreateIntProviderInt() {
+ final long seed = 236784264237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.INT);
+ for (int i = 0; i < 50; i++) {
+ Assertions.assertEquals(rng1.nextInt(), rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderLoHi() {
+ final long seed = 236784264237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.LO_HI);
+ for (int i = 0; i < 50; i++) {
+ final long l = rng1.nextLong();
+ final int hi = (int) (l >>> 32);
+ final int lo = (int) l;
+ Assertions.assertEquals(lo, rng2.nextInt());
+ Assertions.assertEquals(hi, rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderHiLo() {
+ final long seed = 2367234237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.HI_LO);
+ for (int i = 0; i < 50; i++) {
+ final long l = rng1.nextLong();
+ final int hi = (int) (l >>> 32);
+ final int lo = (int) l;
+ Assertions.assertEquals(hi, rng2.nextInt());
+ Assertions.assertEquals(lo, rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderHi() {
+ final long seed = 2367234237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.HI);
+ for (int i = 0; i < 50; i++) {
+ final int hi = (int) (rng1.nextLong() >>> 32);
+ Assertions.assertEquals(hi, rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderLo() {
+ final long seed = 2367234237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.LO);
+ for (int i = 0; i < 50; i++) {
+ final int lo = (int) rng1.nextLong();
+ Assertions.assertEquals(lo, rng2.nextInt());
+ }
+ }
+
+ /**
+ * Test that the default source64 mode matches the nextInt implementation for a LongProvider.
+ * If this changes then the default mode should be updated. The value is used as
+ * the default for the stress test application.
+ */
+ @Test
+ void testCreateIntProviderDefault() {
+ final long seed = 236784264237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), RNGUtils.getSource64Default());
+ for (int i = 0; i < 50; i++) {
+ final int a = rng1.nextInt();
+ final int b = rng1.nextInt();
+ Assertions.assertEquals(a, rng2.nextInt());
+ Assertions.assertEquals(b, rng2.nextInt());
+ }
+ }
+}
diff --git a/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java
index b41a747..9c5d6a0 100644
--- a/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java
+++ b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java
@@ -84,33 +84,54 @@
}
@Test
- void testLongAsIntBigEndian() throws IOException {
+ void testLongAsHLIntBigEndian() throws IOException {
assertRngOutput(RandomSource.SPLIT_MIX_64,
- // Convert SplitMix64 to an int provider so it is detected as requiring double the
+ // Convert to an int provider so it is detected as requiring double the
// length output.
- rng -> new IntProvider() {
- @Override
- public int next() {
- return rng.nextInt();
- }
- },
+ rng -> new HiLoCachingIntProvider(rng),
RngDataOutputTest::writeInt,
- RngDataOutput::ofLongAsInt, ByteOrder.BIG_ENDIAN);
+ RngDataOutput::ofLongAsHLInt, ByteOrder.BIG_ENDIAN);
}
@Test
- void testLongAsIntLittleEndian() throws IOException {
+ void testLongAsHLIntLittleEndian() throws IOException {
assertRngOutput(RandomSource.SPLIT_MIX_64,
// Convert SplitMix64 to an int provider so it is detected as requiring double the
// length output. Then reverse the bytes.
- rng -> new IntProvider() {
+ rng -> new HiLoCachingIntProvider(rng) {
@Override
public int next() {
- return Integer.reverseBytes(rng.nextInt());
+ return Integer.reverseBytes(super.next());
}
},
RngDataOutputTest::writeInt,
- RngDataOutput::ofLongAsInt, ByteOrder.LITTLE_ENDIAN);
+ RngDataOutput::ofLongAsHLInt, ByteOrder.LITTLE_ENDIAN);
+ }
+
+
+ @Test
+ void testLongAsLHIntBigEndian() throws IOException {
+ assertRngOutput(RandomSource.SPLIT_MIX_64,
+ // Convert to an int provider so it is detected as requiring double the
+ // length output.
+ rng -> new LoHiCachingIntProvider(rng),
+ RngDataOutputTest::writeInt,
+ RngDataOutput::ofLongAsLHInt, ByteOrder.BIG_ENDIAN);
+ }
+
+ @Test
+ void testLongAsLHIntLittleEndian() throws IOException {
+ assertRngOutput(RandomSource.SPLIT_MIX_64,
+ // Convert to an int provider so it is detected as requiring double the
+ // length output. Then reverse the bytes.
+ rng -> new LoHiCachingIntProvider(rng) {
+ @Override
+ public int next() {
+ return Integer.reverseBytes(super.next());
+ }
+ },
+ RngDataOutputTest::writeInt,
+ RngDataOutput::ofLongAsLHInt, ByteOrder.LITTLE_ENDIAN);
}
private static void writeInt(DataOutputStream sink, UniformRandomProvider rng) {
@@ -212,4 +233,56 @@
}
return out.toByteArray();
}
+
+ private static class LoHiCachingIntProvider extends IntProvider {
+ private long source = -1;
+ private final UniformRandomProvider rng;
+
+ LoHiCachingIntProvider(UniformRandomProvider rng) {
+ this.rng = rng;
+ }
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.nextLong();
+ // store hi
+ source = next >>> 32;
+ // extract low
+ return (int) next;
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+ }
+
+ private static class HiLoCachingIntProvider extends IntProvider {
+ private long source = -1;
+ private final UniformRandomProvider rng;
+
+ HiLoCachingIntProvider(UniformRandomProvider rng) {
+ this.rng = rng;
+ }
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.nextLong();
+ // store low
+ source = next & 0xffff_ffffL;
+ // extract hi
+ return (int) (next >>> 32);
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+ }
}
diff --git a/commons-rng-examples/examples-stress/stress_test.md b/commons-rng-examples/examples-stress/stress_test.md
index dcd0f31..647a5d7 100644
--- a/commons-rng-examples/examples-stress/stress_test.md
+++ b/commons-rng-examples/examples-stress/stress_test.md
@@ -108,7 +108,7 @@
Test platform Endianness
------------------------
-The stress test application will output raw binary data for generated integers or longs.
+The stress test application will output raw binary data for generated integers or longs.
An integer is 4 bytes and a long is 8 bytes and thus the byte order or [Endianness](https://en.wikipedia.org/wiki/Endianness) of the data
must be correct for the test application. The stress test application can support either big-endian
or little-endian format. The application will auto-detect the platform and will default to output
@@ -170,7 +170,7 @@
| executable | The test tool | dieharder, stdin2testu01, PractRand |
| ... | Arguments for the test tool | dieharder: -a -g 200 -Y 1 -k 2 <br/> stdin2testu01: SmallCrush, Crush, BigCrush <br/> RNG_test stdin32 -tf 1 -te 0 -tlmin 1KB -tlmax 4TB |
-The `stress` command defaults for other options that can be changed, for example the output
+The `stress` command defaults for other options can be changed, for example the output
results file prefix, the number of concurrent tasks, byte-order or the number of trials per
generator.
Use the `--help` option to show the available options.
@@ -250,24 +250,42 @@
java -jar target/examples-stress.jar results \
target/tu_* \
target/dh_* \
- target/pr_* \
+ target/pr_*
Various formats are available. Use the `--help` option to show the available options.
Test 64-bit generators
----------------------
-The available random generators output either 32-bits or 64-bits per cycle.
+The available random generators output either 32-bits or 64-bits per cycle.
Any application that supports 64-bit generators and should be tested using the `--raw64`
output mode of the `stress` command. See the example for running tests using **PractRand**
for details.
Any application that supports 32-bit generators can be tested using different subsets of the
-64-bit output. For example the test applications **Dieharder** and **TestU01** require 32-bit input.
-The standard method for a 64-bit generator is to use the upper and then lower 32-bits of each
-64-bit output. The stress test application has options to use only the upper or the lower 32-bits
-for testing. These can then be bit-reversed or byte-reversed if desired.
+64-bit output using the `--source64` option. For example the test applications **Dieharder**
+and **TestU01** require 32-bit input.
+The standard method for a 64-bit generator is to use the full 64-bits of output in the order
+provided by the Commons RNG core caching implementation of the `nextInt` method. The stress
+test application has options to use the following parts of the 64-bit output:
+
+| source64 | Description |
+| --------- | ---------- |
+| LONG | Use the full 64-bit output. This is equivalent to using the `--raw64` mode. |
+| INT | Use the 32-bit output from the RNG implementation of nextInt. |
+| HI_LO | Use the upper 32-bits, then lower 32-bits of the 64-bit output. |
+| LO_HI | Use the lower 32-bits, then upper 32-bits of the 64-bit output. |
+| HI | Use the upper 32-bits of the 64-bit output. |
+| LO | Use the lower 32-bits of the 64-bit output. |
+
+Note: The default uses an explicit mode rather than INT so that it is recorded in the stress
+test output file. If the RNG uses the default implementation for `nextInt` then output
+from INT should match LO_HI.
+
+On some platforms the LONG setting is matched by a split of the upper and lower parts.
+The binary output from LONG should match LO_HI if using little-endian format;
+it should match HI_LO if using big-endian format.
The `list` command can output available generators by provider type. For example to output the
64-bit providers to a `rng64.list` file:
@@ -280,11 +298,14 @@
java -jar target/examples-stress.jar stress \
--list rng64.list \
- --low-bits \
+ --source64 LO \
--reverse-bits \
--prefix target/tu_lo_r_ \
./stdin2testu01 \
BigCrush
-If a 32-bit provider is used with the `--low-bits` or `--upper-bits` options then an error
-message is shown.
+If a 32-bit provider is used with the `--source64=LONG` or `--raw64` options then
+an error message is shown as this is explicitly for 64-bit output. Other `source64`
+options split the 64-bit output for testing with a 32-bit stress test application. These
+options are ignored for any 32-bit provider; this allows testing 64-bit and 32-bit
+providers in the same batch execution.
