MINOR: Bump org.apache.parquet:parquet-variant from 1.17.1 to 1.18.1 (#1287) Bumps [org.apache.parquet:parquet-variant](https://github.com/apache/parquet-java) from 1.17.1 to 1.18.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/apache/parquet-java/releases">org.apache.parquet:parquet-variant's releases</a>.</em></p> <blockquote> <h2>Apache Parquet 1.18.1 RC1</h2> <p><strong>Full Changelog</strong>: <a href="https://github.com/apache/parquet-java/compare/apache-parquet-1.18.0...apache-parquet-1.18.1-rc1">https://github.com/apache/parquet-java/compare/apache-parquet-1.18.0...apache-parquet-1.18.1-rc1</a></p> <h2>Apache Parquet 1.18.0</h2> <h2>What's Changed</h2> <ul> <li>MINOR: Fix error message in SimpleGroup.add() for Binary by <a href="https://github.com/takezoe"><code>@takezoe</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3385">apache/parquet-java#3385</a></li> <li>[minor] remove comment from pom.xml by <a href="https://github.com/dossett"><code>@dossett</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3382">apache/parquet-java#3382</a></li> <li>Minor: <code>ci-before_install.sh</code> should show wget errors and retry by <a href="https://github.com/dossett"><code>@dossett</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3381">apache/parquet-java#3381</a></li> <li>Minor: Upgrade spotless plugin by <a href="https://github.com/dossett"><code>@dossett</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3380">apache/parquet-java#3380</a></li> <li>[minor] Move on from java8 for three core modules by <a href="https://github.com/dossett"><code>@dossett</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3379">apache/parquet-java#3379</a></li> <li>MINOR: <code>japicmp</code> checks are failing after Java 11 release by <a href="https://github.com/dossett"><code>@dossett</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3388">apache/parquet-java#3388</a></li> <li>MINOR: Bump version to 1.18.0-SNAPSHOT by <a href="https://github.com/wgtmac"><code>@wgtmac</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3389">apache/parquet-java#3389</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3372">GH-3372</a>: Enhance ColumnSizeCommand to sort by size by <a href="https://github.com/wgtmac"><code>@wgtmac</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3371">apache/parquet-java#3371</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3399">GH-3399</a>: Hint IDE the source code generated by thrift by <a href="https://github.com/HuaHuaY"><code>@HuaHuaY</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3400">apache/parquet-java#3400</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3387">GH-3387</a>: bump aircompressor to mitigate CVE-2025-67721 by <a href="https://github.com/olegbonar"><code>@olegbonar</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3403">apache/parquet-java#3403</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3383">GH-3383</a>: Modernize mockito by <a href="https://github.com/dossett"><code>@dossett</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3384">apache/parquet-java#3384</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3404">GH-3404</a>: fix CI TestByteBitPacking512VectorLE.unpackValuesUsingVector randomly oom by <a href="https://github.com/HuaHuaY"><code>@HuaHuaY</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3405">apache/parquet-java#3405</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3401">GH-3401</a>: Remove redundant MAP_KEY_VALUE annotation at middle level of logical type map by <a href="https://github.com/HuaHuaY"><code>@HuaHuaY</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3402">apache/parquet-java#3402</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3406">GH-3406</a>: fix Float16 statistics handling for NaN and zero values by <a href="https://github.com/HuaHuaY"><code>@HuaHuaY</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3407">apache/parquet-java#3407</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3215">GH-3215</a>: Sort the list of org.apache.parquet.format.Encoding created … by <a href="https://github.com/mbrannstrom"><code>@mbrannstrom</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3413">apache/parquet-java#3413</a></li> <li>Bump dependabot limit by <a href="https://github.com/Fokko"><code>@Fokko</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3416">apache/parquet-java#3416</a></li> <li>Bump org.mockito:mockito-core from 5.21.0 to 5.22.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3436">apache/parquet-java#3436</a></li> <li>Bump org.apache:apache from 33 to 37 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3431">apache/parquet-java#3431</a></li> <li>Bump protobuf.version from 4.33.2 to 4.34.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3441">apache/parquet-java#3441</a></li> <li>Bump com.github.luben:zstd-jni from 1.5.7-3 to 1.5.7-7 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3442">apache/parquet-java#3442</a></li> <li>Bump org.apache.maven.plugins:maven-shade-plugin from 3.6.0 to 3.6.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3430">apache/parquet-java#3430</a></li> <li>Bump org.xerial.snappy:snappy-java from 1.1.10.7 to 1.1.10.8 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3426">apache/parquet-java#3426</a></li> <li>Bump org.codehaus.mojo:exec-maven-plugin from 3.5.0 to 3.6.3 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3440">apache/parquet-java#3440</a></li> <li>Bump org.apache.parquet:parquet-pig from 1.15.0 to 1.15.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3420">apache/parquet-java#3420</a></li> <li>Bump commons-logging:commons-logging from 1.3.5 to 1.3.6 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3418">apache/parquet-java#3418</a></li> <li>Bump com.google.re2j:re2j from 1.7 to 1.8 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3433">apache/parquet-java#3433</a></li> <li>Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.20.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3429">apache/parquet-java#3429</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3411">GH-3411</a> Expose row group index via Parquet reader by <a href="https://github.com/uros7251brick"><code>@uros7251brick</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3412">apache/parquet-java#3412</a></li> <li>Bump org.mockito:mockito-core from 5.22.0 to 5.23.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3445">apache/parquet-java#3445</a></li> <li>Bump net.openhft:zero-allocation-hashing from 0.27ea0 to 0.27ea1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3419">apache/parquet-java#3419</a></li> <li>Bump org.tukaani:xz from 1.10 to 1.12 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3417">apache/parquet-java#3417</a></li> <li>Bump it.unimi.dsi:fastutil from 8.5.15 to 8.5.18 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3422">apache/parquet-java#3422</a></li> <li>Bump org.apache.commons:commons-text from 1.13.1 to 1.15.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3438">apache/parquet-java#3438</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3449">GH-3449</a>: Fix BinaryStats int overflow by <a href="https://github.com/Jiayi-Wang-db"><code>@Jiayi-Wang-db</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3448">apache/parquet-java#3448</a></li> <li>Bump protobuf.version from 4.34.0 to 4.34.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3453">apache/parquet-java#3453</a></li> <li>Bump org.apache.arrow:arrow-vector from 17.0.0 to 19.0.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3454">apache/parquet-java#3454</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3410">GH-3410</a>: Bump jackson dependencies from 2.19.2 to 2.21.2 by <a href="https://github.com/manuzhang"><code>@manuzhang</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3460">apache/parquet-java#3460</a></li> <li>Bump com.github.siom79.japicmp:japicmp-maven-plugin from 0.23.1 to 0.25.4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3437">apache/parquet-java#3437</a></li> <li>Bump com.github.siom79.japicmp:japicmp-maven-plugin from 0.25.4 to 0.25.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3473">apache/parquet-java#3473</a></li> <li>Bump com.github.siom79.japicmp:japicmp-maven-plugin from 0.25.5 to 0.25.6 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3491">apache/parquet-java#3491</a></li> <li>Bump com.google.guava:guava from 33.5.0-jre to 33.6.0-jre by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/apache/parquet-java/pull/3492">apache/parquet-java#3492</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3011">GH-3011</a>: Deny further writes after InternalParquetRecordWriter is aborted by <a href="https://github.com/LuciferYang"><code>@LuciferYang</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3450">apache/parquet-java#3450</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3461">GH-3461</a>: Avoid unnecessary debug logging overhead in record write hot path by <a href="https://github.com/arouel"><code>@arouel</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3462">apache/parquet-java#3462</a></li> <li><a href="https://redirect.github.com/apache/parquet-java/issues/3408">GH-3408</a> Add support for stale PR workflow by <a href="https://github.com/ArnavBalyan"><code>@ArnavBalyan</code></a> in <a href="https://redirect.github.com/apache/parquet-java/pull/3409">apache/parquet-java#3409</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/apache/parquet-java/blob/master/CHANGES.md">org.apache.parquet:parquet-variant's changelog</a>.</em></p> <blockquote> <!-- raw HTML omitted --> <h1>Parquet</h1> <p>From 1.14.2 onwards, the Parquet project has migrated from Jira to GitHub, and the changelogs are now <a href="https://github.com/apache/parquet-java/releases">published under Releases</a>.</p> <h3>Version 1.14.1</h3> <p>Release Notes - Parquet - Version 1.14.1</p> <h4>Bug</h4> <ul> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2468">PARQUET-2468</a> - ParquetMetadata.toPrettyJSON throws exception on file read when LOG.isDebugEnabled()</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2498">PARQUET-2498</a> - Hadoop vector IO API doesn't handle empty list of ranges</li> </ul> <h3>Version 1.14.0</h3> <p>Release Notes - Parquet - Version 1.14.0</p> <h4>Bug</h4> <ul> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2260">PARQUET-2260</a> - Bloom filter bytes size shouldn't be larger than maxBytes size in the configuration</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2266">PARQUET-2266</a> - Fix support for files without ColumnIndexes</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2276">PARQUET-2276</a> - ParquetReader reads do not work with Hadoop version 2.8.5</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2300">PARQUET-2300</a> - Update jackson-core 2.13.4 to a version without CVE PRISMA-2023-0067</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2325">PARQUET-2325</a> - Fix parquet-cli's dictionary subcommand to work with FIXED_LEN_BYTE_ARRAY</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2329">PARQUET-2329</a> - Fix wrong help messages of parquet-cli subcommands</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2330">PARQUET-2330</a> - Fix convert-csv to show the correct position of the invalid record</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2332">PARQUET-2332</a> - Fix unexpectedly disabled tests to be executed</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2336">PARQUET-2336</a> - Add caching key to CodecFactory</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2342">PARQUET-2342</a> - Parquet writer produced a corrupted file due to page value count overflow</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2343">PARQUET-2343</a> - Fixes NPE when rewriting file with multiple rowgroups</li> <li><a href="https://issues.apache.org/jira/browse/PARQUET-2348">PARQUET-2348</a> - Recompression/Re-encrypt should rewrite bloomfilter</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/apache/parquet-java/commit/7c22cdba6792fa29c5bda74bf1f866a08d7572a9"><code>7c22cdb</code></a> [maven-release-plugin] prepare release apache-parquet-1.18.1-rc1</li> <li><a href="https://github.com/apache/parquet-java/commit/4c04bf78c01e62ebc26e355e4995790acc89a2e3"><code>4c04bf7</code></a> Prepare for next development iteration</li> <li><a href="https://github.com/apache/parquet-java/commit/4ae935328b030c9aa116bb6c0fd166ef5c66cd2f"><code>4ae9353</code></a> Bump jackson.version from 2.22.1 to 2.22.2 (<a href="https://redirect.github.com/apache/parquet-java/issues/3739">#3739</a>)</li> <li><a href="https://github.com/apache/parquet-java/commit/a3501b61622be52c9620c0ed93480417f27b5600"><code>a3501b6</code></a> <code>BytesInput.copy(BytesInput)</code> should return a copy (<a href="https://redirect.github.com/apache/parquet-java/issues/3729">#3729</a>)</li> <li><a href="https://github.com/apache/parquet-java/commit/3fd0ae7684e5768723158fad296db1894a633e55"><code>3fd0ae7</code></a> Add parameter to enable reading KMS URL from key material (<a href="https://redirect.github.com/apache/parquet-java/issues/3745">#3745</a>)</li> <li><a href="https://github.com/apache/parquet-java/commit/afe34d8b04d82b6dace63901074d4d5f8e6b54e3"><code>afe34d8</code></a> Bump com.fasterxml.jackson.core:jackson-databind from 2.22.1 to 2.22.2 (<a href="https://redirect.github.com/apache/parquet-java/issues/3740">#3740</a>)</li> <li><a href="https://github.com/apache/parquet-java/commit/1916eff07f1315003a59d55e24b97919e741f329"><code>1916eff</code></a> Bump com.github.luben:zstd-jni from 1.5.7-13 to 1.5.7-15 (<a href="https://redirect.github.com/apache/parquet-java/issues/3744">#3744</a>)</li> <li><a href="https://github.com/apache/parquet-java/commit/b1b7ffcad5e72d8dd80905290a83d021e56332c5"><code>b1b7ffc</code></a> Bump com.github.luben:zstd-jni from 1.5.7-12 to 1.5.7-13 (<a href="https://redirect.github.com/apache/parquet-java/issues/3725">#3725</a>)</li> <li><a href="https://github.com/apache/parquet-java/commit/686bb10c93b9b4c3b9e106715f32f29ad2d47504"><code>686bb10</code></a> Bump it.unimi.dsi:fastutil from 8.5.18 to 8.5.19 (<a href="https://redirect.github.com/apache/parquet-java/issues/3684">#3684</a>)</li> <li><a href="https://github.com/apache/parquet-java/commit/663e9074c424262873cc8d35ead16da2c581522c"><code>663e907</code></a> Bump com.github.luben:zstd-jni from 1.5.7-11 to 1.5.7-12 (<a href="https://redirect.github.com/apache/parquet-java/issues/3704">#3704</a>)</li> <li>Additional commits viewable in <a href="https://github.com/apache/parquet-java/compare/apache-parquet-1.17.1...apache-parquet-1.18.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: JB Onofré <jbonofre@apache.org>
The following guides explain the fundamental data structures used in the Java implementation of Apache Arrow.
Generated javadoc documentation is available here.
Refer to Building Apache Arrow for documentation of environment setup and build instructions.
Arrow uses Google's Flatbuffers to transport metadata. The java version of the library requires the generated flatbuffer classes can only be used with the same version that generated them. Arrow packages a version of the arrow-vector module that shades flatbuffers and arrow-format into a single JAR. Using the classifier “shade-format-flatbuffers” in your pom.xml will make use of this JAR, you can then exclude/resolve the original dependency to a version of your choosing.
$ flatc --version flatc version 25.1.24 $ grep "dep.fbs.version" pom.xml <dep.fbs.version>25.1.24</dep.fbs.version>
cd $ARROW_HOME # remove the existing files rm -rf format/src # regenerate from the .fbs files flatc --java -o format/src/main/java arrow-format/*.fbs # prepend license header mvn spotless:apply -pl :arrow-format
There are several system/environmental variables that users can configure. These trade off safety (they turn off checking) for speed. Typically they are only used in production settings after the code has been thoroughly tested without using them.
Bounds Checking for memory accesses: Bounds checking is on by default. You can disable it by setting either the system property(arrow.enable_unsafe_memory_access) or the environmental variable (ARROW_ENABLE_UNSAFE_MEMORY_ACCESS) to true. When both the system property and the environmental variable are set, the system property takes precedence.
null checking for gets: ValueVector get methods (not getObject) methods by default verify the slot is not null. You can disable it by setting either the system property(arrow.enable_null_check_for_get) or the environmental variable (ARROW_ENABLE_NULL_CHECK_FOR_GET) to false. When both the system property and the environmental variable are set, the system property takes precedence.
-Dio.netty.tryReflectionSetAccessible=true should be set. This fixes java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available. thrown by Netty.StructVector enable -Darrow.struct.conflict.policy=CONFLICT_APPEND. Duplicate fields are ignored (CONFLICT_REPLACE) by default and overwritten. To support different policies for conflicting or duplicate fields set this JVM flag or use the correct static constructor methods for StructVectors.Arrow Java follows the Google Java Style Guide with the following differences:
NoFinalizer, OverloadMethodsDeclarationOrder, and VariableDeclarationUsageDistance due to the existing code base. These rules should be followed when possible.Refer to checkstyle.xml for rule specifics.
When running tests, Arrow Java uses the Logback logger with SLF4J. By default, it uses the logback.xml present in the corresponding module's src/test/resources directory, which has the default log level set to INFO. Arrow Java can be built with an alternate logback configuration file using the following command run in the project root directory:
mvn -Dlogback.configurationFile=file:<path-of-logback-file>
See Logback Configuration for more details.
Integration tests which require more time or more memory can be run by activating the integration-tests profile. This activates the Maven Failsafe plugin and any class prefixed with IT will be run during the testing phase. The integration tests currently require a larger amount of memory (>4GB) and time to complete. To activate the profile:
mvn -Pintegration-tests <rest of mvn arguments>