| commit | da4c769b2e21b21f0f981cc88c0d068cce6a7f45 | [log] [tgz] |
|---|---|---|
| author | Lee Rhodes <leerho@users.noreply.github.com> | Mon May 09 17:25:08 2022 -0700 |
| committer | Lee Rhodes <leerho@users.noreply.github.com> | Mon May 09 17:25:08 2022 -0700 |
| tree | f98ce1064dda17d411b5205cda64637bd1670746 | |
| parent | b4562150ce1cf5ea44386c9090a58c5342b70c4c [diff] |
Spelling
=================
This Memory component is general purpose, has no external runtime dependencies and can be used in any application that needs to manage data structures inside or outside the Java heap.
The goal of this component of the DataSketches library is to provide a high performance access API for accessing four different types of memory resources. Each of the four resource types is accessed using different API methods in the Memory component.
Heap: Contiguous bytes on the Java Heap constructed by, e.g., WritableMemory.writableWrap(byte[]) or using the WritableMemory.allocate(int) method. For purposes of this document this includes on-heap ByteBuffers constructed using ByteBuffer.allocate(int).
DirectByteBuffer: Contiguous bytes of a ByteBuffer constructed by, e.g., WritableMemory.writableWrap(ByteBuffer) where the ByteBuffer was previously constructed using ByteBuffer.allocateDirect(int); or, is a slice() thereof.
Direct: Contiguous bytes off the Java Heap constructed by, e.g., WritableMemory.allocateDirect(long) method.
Memory-Mapped Files Contiguous bytes of a file represented in off-heap memory and created using, e.g., the WritableMemory.writableMap(File) method.
Please visit the main DataSketches website for more information.
If you are interested in making contributions to this Memory component please see our Community page.
Starting with release datasketches-memory-2.0.0, this Memory component supports Java 8 through Java 13. Providing access to the four contiguous byte resources (mentioned above) in Java 8 only requires reflection. However, Java 9 introduced the Java Platform Module System (JPMS) where access to these internal classes requires starting up the JVM with special JPMS arguments. The actual JVM arguments required will depend on how the user intends to use the Memory API, the Java version used to run the user‘s application and whether the user’s application is a JPMS application or not.
Also see the usage examples for more information.
In this environment, the user is using the Jars from Maven Central as a library dependency and not attempting to build the Memory component from the source code or run the Memory component tests.
Otherwise, if you are running Java 9-13 and ...
| JVM Arguments for non-JPMS Applications | Direct ByteBuffer | Direct | MemoryMapped Files |
|---|---|---|---|
| --add-exports java.base/jdk.internal.misc= ALL-UNNAMED | Yes | ||
| --add-exports java.base/jdk.internal.ref= ALL-UNNAMED | Yes | Yes | |
| --add-opens java.base/java.nio= ALL-UNNAMED | Yes | Yes | |
| --add-opens java.base/sun.nio.ch= ALL-UNNAMED | Yes |
| JVM Arguments for JPMS Applications | Direct ByteBuffer | Direct | MemoryMapped Files |
|---|---|---|---|
| --add-exports java.base/jdk.internal.misc= org.apache.datasketches.memory | Yes | ||
| --add-exports java.base/jdk.internal.ref= org.apache.datasketches.memory | Yes | Yes | |
| --add-opens java.base/java.nio= org.apache.datasketches.memory | Yes | Yes | |
| --add-opens java.base/sun.nio.ch= org.apache.datasketches.memory | Yes | Yes |
In this environment the developer needs to build the Memory component from source and run the Memory Component tests. There are two use cases. The first is for a System Developer that needs to build and test their own Jar from source for a specific Java version. The second use case is for a Memory Component Developer and Contributor.
System Developer
Memory Component Developer / Contributor
NOTES:
This component accesses resource files for testing. As a result, the directory elements of the full absolute path of the target installation directory must qualify as Java identifiers. In other words, the directory elements must not have any space characters (or non-Java identifier characters) in any of the path elements. This is required by the Oracle Java Specification in order to ensure location-independent access to resources: See Oracle Location-Independent Access to Resources
This project is structured as a Maven multi-module project. Building this project might affect plugins that require early dependency resolution, such as the javadoc and eclipse plugins. The Maven build instructions below have been modified to use the process-classes phase (instead of compile) for these use cases.
For more information, see this Maven Reactor Issue.
There are no run-time dependencies. See the pom.xml file for test dependencies.
The Maven build requires the following JDKs to compile:
Before building, first ensure that your local environment has been configured according to the Maven Toolchains Configuration.
There are three types of tests: normal unit tests, tests run by the strict profile and continuous integration(CI) tests. The CI tests target the Multi-Release (MR) JAR and run the entire test suite using a specific version of Java. Running the CI test command also runs the default unit tests.
To run normal unit tests:
mvn clean test
To run the strict profile tests (only supported in Java 8):
mvn clean test -P strict
To run javadoc on this multi-module project, use:
mvn clean process-classes javadoc:javadoc -DskipTests=true
To run the CI tests against the multi-release JAR for specific JVM versions [9-13], use:
mvn clean package -Denvironment=ci -Dmatrix.jdk.version=9
To run the eclipse plugin on this multi-module project, use:
mvn clean process-classes eclipse:eclipse -DskipTests=true
To install jars built from the downloaded source:
mvn clean install -DskipTests=true
This will create the following Jars:
A build script named compile-package-jar.sh has been provided to package a JAR for a specific java version. This is necessary in cases where a developer is unable to install all the required versions of the JDK that are required as part of the Maven build.
The build script performs the following steps:
The build script is located in the tools/scripts/ directory and requires the following arguments:
For example, if the project base directory is /src/datasketches-memory;
To run the script for a release version:
./tools/scripts/compile-package-jar.sh $JAVA_HOME 2.0.0 /src/datasketches-memory
To run the script for a snapshot version:
./tools/scripts/compile-package-jar.sh $JAVA_HOME 2.1.0-SNAPSHOT /src/datasketches-memory
To run the script for an RC version:
./tools/scripts/compile-package-jar.sh $JAVA_HOME 2.0.0-RC1 /src/datasketches-memory
Note that the script does not use the Git Version Tag to adjust the working copy to a remote tag - it is expected that the user has a pristine copy of the desired branch/tag available before using the script.
For more information on the project configuration, the following topics are discussed in more detail:
In order to build and contribute to this project, please read the relevant IDE documentation: