rocketmq-rocksdb releases are a fat jar file that contain the following binaries:
To build the binaries for a rocketmq-rocksdb release, follow the steps below. 1.git clone https://github.com/facebook/rocksdb.git
2.copy RemoveConsumeQueueCompactionFilter.java to rocksdb/java/src/main/java/org/rocksdb/ 3.copy remove_consumequeue_compactionfilterjni.cc to rocksdb/java/rocksjni/ 4.copy remove_consumequeue_compactionfilter.h to rocksdb/utilities/compaction_filters/ 5.copy remove_consumequeue_compactionfilter.cc to rocksdb/utilities/compaction_filters/
6.replace rocksdb/thirdparty.inc with thirdparty.inc with
7.add the string “rocksjni/remove_consumequeue_compactionfilterjni.cc” to the file rocksdb/java/CMakeLists.txt 8.add the string “src/main/java/org/rocksdb/RemoveConsumeQueueCompactionFilter.java” to the file rocksdb/java/CMakeLists.txt 9.add the string “org.rocksdb.RemoveConsumeQueueCompactionFilter” to the file rocksdb/java/CMakeLists.txt 10.add the string "org.rocksdb.RemoveConsumeQueueCompactionFilter" to the file rocksdb/java/Makefile 11.add the string "addutilities/compaction_filters/remove_consumequeue_compactionfilter.cc " to the file rocksdb/src.mk 12.add the string "java/rocksjni/remove_consumequeue_compactionfilterjni.cc " to the file rocksdb/src.mk 13.add the string ‘“utilities/compaction_filters/remove_consumequeue_compactionfilter.cc”,’ to the file rocksdb/TARGETS
14.change ‘set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS} /WX”)’ in file rocksdb/CMakeLists.txt to ‘set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS}”)’ 15.change ‘set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS} -Werror”)’ in file rocksdb/CMakeLists.txt to ‘set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS}”)’
16.change ‘DEBUG_LEVEL?=1’ to ‘DEBUG_LEVEL?=0’ in file rocksdb/Makefile
Both mac compilation and linux compilation are performed on the mac system, but we finish compilation on linux by using the docker image provided by rocksdb. There is one more thing to note here, the docker images downloaded in the Makefile execution script may be out of date due to its jdk version, resulting in failure in the end. For example, the docker image evolvedbinary/rocksjava:centos6_x86-be was last updated 4 years ago, and the system image uses jdk1.7, while the minimum jdk required for the current rocksdb compilation is 1.8.
Fortunately, rocksdb has its own docker warehouse, which can build a local image by itself, which can avoid downloading images that have not been updated for a long time from the official website of rocksdb. The warehouse address is as follows: https://github.com/evolvedbinary/docker-rocksjava
git clone https://github.com/evolvedbinary/docker-rocksjava
You can compile the docker images required in the Makefile by yourself, and the compilation script is in the README file. In addition, it may be necessary to modify the url in the Dockerfile of some operating systems,
http://mirrors.aliyun.com/alpine/v3.13/main" > /etc/apk/repositories \ && echo "http://mirrors.aliyun.com/alpine/v3.13/community" >> /etc/apk/repositories \ && apk update \ && apk update
It is very important to make the linux docker image by yourself. If you let the RocksDB compilation script download the linux compilation environment, not only will the process be very slow, but the downloaded linux compilation environment may even be out of date. So please carefully read the readme of the docker-rocksjava warehouse and make the linux docker image by yourself. The linux image we need is as follows:
apline3_x86-be apline3_x64-be centos6_x64-be centos6_x86-be centos7_ppc64le-be
After all linux docker images are ready, run this make command from RocksDB's root source directory:
make jclean clean rocksdbjavastaticreleasedocker
This command will build RocksDB natively on OSX, and will then spin up docker containers to build RocksDB for 32-bit and 64-bit Linux with glibc, and 32-bit and 64-bit Linux with musl libc.
You can find all native binaries and JARs in the java/target directory upon completion:
librocksdbjni-linux32.so librocksdbjni-linux64.so librocksdbjni-linux64-musl.so librocksdbjni-linux32-musl.so librocksdbjni-osx.jnilib rocksdbjni-x.y.z-javadoc.jar rocksdbjni-x.y.z-linux32.jar rocksdbjni-x.y.z-linux64.jar rocksdbjni-x.y.z-linux64-musl.jar rocksdbjni-x.y.z-linux32-musl.jar rocksdbjni-x.y.z-osx.jar rocksdbjni-x.y.z-sources.jar rocksdbjni-x.y.z.jar
Where x.y.z is the built version number of RocksDB.
And what we need is:
rocksdbjni-x.y.z-javadoc.jar rocksdbjni-x.y.z-sources.jar rocksdbjni-x.y.z.jar
The Windows system needs to be compiled independently, please follow the steps below. Need to install:
Unlike linux or mac systems, the compilation script will automatically download and compile several compression algorithm warehouses like snappy, lz4. In the Windows environment, you need to download and compile by yourself. For details, please refer to the document:
http://rocksdb.org.cn/doc/Building-on-Windows.html
Finally, run this make command from RocksDB's root source directory:
set THIRDPARTY_HOME=D:\rocketmq-rocksdb set MSBUILD=C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20 -DJNI=1 -DSNAPPY=1 -DLZ4=1 -DZLIB=1 -DZSTD=1 -DXPRESS=1 .. "%MSBUILD%" rocksdb.sln /p:Configuration=Release
The resulting native binary will be built and available at build\java\Release\rocksdbjni-shared.dll
. You can also find it under project folder with name librocksdbjni-win64.dll
.
librocksdbjni-win64.dll
to rocksdbjni-x.y.z.jar
jar -uf rocksdbjni-x.y.z.jar librocksdbjni-win64.dll
Finally, we use the following file to release an official rocksdb release for rocketmq:
rocksdbjni-x.y.z-javadoc.jar rocksdbjni-x.y.z-sources.jar rocksdbjni-x.y.z.jar