ARROW-7580: [Website] 0.16 release post (#41)
* Create outline for 0.16 release post
* Add release notes
Add highlights for C++, Python, community discussions.
* Add dataset changes
* Add Rust notes
* Add Ruby and C GLib notes
* Add syntax highlighting
* update Rust notes
* Add some notes, simplify some of hte C++ section for readability
* Revise/extend 0.16 blog post
* Add some update fo specification and Java
* Added a java bug fix bullet point.
* add python 2.7 note
* Fix typo
* Rename file
* Edits for publication
* More cleanup
* Update publication date
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: François Saint-Jacques <fsaintjacques@gmail.com>
Co-authored-by: Wakahisa <nevilledips@gmail.com>
Co-authored-by: Yosuke Shiro <yosuke.shiro615@gmail.com>
Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
Co-authored-by: emkornfield <emkornfield@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Kenta Murata <mrkn@users.noreply.github.com>
diff --git a/_posts/2020-02-11-0.16.0-release.md b/_posts/2020-02-11-0.16.0-release.md
new file mode 100644
index 0000000..e7acd3c
--- /dev/null
+++ b/_posts/2020-02-11-0.16.0-release.md
@@ -0,0 +1,292 @@
+---
+layout: post
+title: "Apache Arrow 0.16.0 Release"
+date: "2020-02-12 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+The Apache Arrow team is pleased to announce the 0.16.0 release. This covers
+about 4 months of development work and includes [**735 resolved issues**][1]
+from [**99 distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## New committers
+
+Since the 0.15.0 release, we've added two new committers:
+
+* [Eric Erhardt][4]
+* [Joris Van den Bossche][5]
+
+Thank you for all your contributions!
+
+## Columnar Format Notes
+
+We still have work to do to complete comprehensive columnar format integration
+testing between the Java and C++ libraries. Once this work is completed, we
+intend to make a 1.0.0 release with [forward and backward compatibility
+guarantees][23].
+
+We [clarified some ambiguity][27] on dictionary encoding in the specification.
+Work is on going to implement the features in Arrow libraries.
+
+## Arrow Flight RPC notes
+
+Flight development work has recently focused on robustness and stability. If
+you are not yet familiar with Flight, read the [introductory blog post from
+October][24].
+
+We are also discussing adding a "bidirectional RPC" which enables
+request-response workflows requiring both client and server to send data
+streams to be performed a single RPC request.
+
+## C++ notes
+
+Some work has been done to make the default build configuration of Arrow C++ as
+lean as possible. The Arrow C++ core can now be built without any external
+dependencies other than a new enough C++ compiler (gcc 4.9 or higher). Notably,
+Boost is no longer required. We invested effort to vendor some small essential
+dependencies: Flatbuffers, double-conversion, and uriparser. Many optional
+features requiring external libraries, like compression and GLog integration,
+are now disabled by default. Several subcomponents of the C++ project like the
+filesystem API, CSV, compute, dataset and JSON layers, as well as command-line
+utilities, are now disabled by default. The only toolchain dependency enabled
+by default is jemalloc, the recommended memory allocator, but this can also be
+disabled if desired. For illustration, see the [example minimal build script and
+Dockerfile][28].
+
+When enabled, the default jemalloc configuration has been tweaked to return
+memory more aggressively to the OS (ARROW-6910, ARROW-6994). We welcome
+feedback from users about our memory allocation configuration and performance
+in applications.
+
+The array validation facilities have been vastly expanded and now exist in
+two flavors: the `Validate` method does a light-weight validation that's
+O(1) in array size, while the potentially O(N) method `ValidateFull` does
+thorough data validation (ARROW-6157).
+
+The IO APIs now use `Result<T>` when returning both a Status
+and result value, rather than taking a pointer-out function parameter
+(ARROW-7235).
+
+### C++: CSV
+
+An option is added to attempt automatic dictionary encoding of string columns
+during reading a CSV file, until a cardinality limit is reached. When
+successful, it can make reading faster and the resulting Arrow data is
+much more memory-efficient (ARROW-3408).
+
+The CSV APIs now use `Result<T>` when returning both a Status
+and result value, rather than taking a pointer-out function parameter
+(ARROW-7236).
+
+### C++: Datasets
+
+The 0.16 release introduces the Datasets API to the C++ library, along with
+bindings in Python and R. This API allows you to treat multiple files as a
+single logical dataset entity and make efficient selection queries against it.
+This release includes support for Parquet and Arrow IPC file formats. Factory
+objects allow you to discover files in a directory recursively, inspect the
+schemas in the files, and performs some basic schema unification. You may
+specify how file path segments map to partition, and there is support for
+auto-detecting some partition information, including Hive-style partitioning.
+The Datasets API includes a filter expression syntax as well as column
+selection. These are evaluated with predicate pushdown, and for Parquet,
+evaluation is pushed down to row groups.
+
+### C++: Filesystem layer
+
+An HDFS implementation of the FileSystem class is available (ARROW-6720). We
+plan to deprecate the prior bespoke C++ HDFS class in favor of the standardized
+filesystem API.
+
+The filesystem APIs now use `Result<T>` when returning both a Status
+and result value, rather than taking a pointer-out function parameter
+(ARROW-7161).
+
+### C++: IPC
+
+The Arrow IPC reader is being fuzzed continuously by the [OSS-Fuzz][6]
+infrastructure, to detect undesirable behavior on invalid or malicious input.
+Several issues have already been found and fixed.
+
+### C++: Parquet
+
+[Modular encryption][10] is now supported (PARQUET-1300).
+
+A performance regression when reading a file with a large number of columns has
+been fixed (ARROW-6876, ARROW-7059), as well as several bugs (PARQUET-1766,
+ARROW-6895).
+
+### C++: Tensors
+
+CSC sparse matrices are supported (ARROW-4225).
+
+The Tensor APIs now use `Result<T>` when returning both a Status
+and result value, rather than taking a pointer-out function parameter
+(ARROW-7420).
+
+## C# Notes
+
+There were a number of C# bug fixes this release. Note that the C# library is
+not yet being tested in CI against the other native Arrow implementations
+(integration tests). We are looking for more contributors for the C# project to
+help with this and other new feature development.
+
+## Java notes
+
+* Added prose documentation describing how to work with the Java libraries.
+* Some additional algorithms have been added to the "contrib" algorithms package: multithreaded searching of ValueVectors,
+* The memory modules have been refactored so non-netty allocators can now be used.
+* A new utility for populating ValueVectors more concisely for testing was introduced
+* The "contrib" Avro adapter now supports Avro Logical type conversion to corresponding Arrow type.
+* Various bug fixes across all packages
+
+## Python notes
+
+pyarrow 0.16 will be the last release to support Python 2.7.
+
+Python now has bindings for the datasets API (ARROW-6341) as well as the S3
+(ARROW-6655) and HDFS (ARROW-7310) filesystem implementations.
+
+The Duration (ARROW-5855) and Fixed Size List (ARROW-7261) types are exposed
+in Python.
+
+Sparse tensors can be converted to dense tensors (ARROW-6624). They are
+also interoperable with the `pydata/sparse` and `scipy.sparse` libraries
+(ARROW-4223, ARROW-4224).
+
+Pandas extension arrays now are able to roundtrip through Arrow conversion
+(ARROW-2428).
+
+A memory leak when converting Arrow data to Pandas "object" data has been
+fixed (ARROW-6874).
+
+Arrow is now tested against Python 3.8, and we now build manylinux2014 wheels
+for Python 3 (ARROW-7344).
+
+## R notes
+
+This release includes a `dplyr` interface to Arrow Datasets,
+which let you work efficiently with large, multi-file datasets as a single entity.
+See [`vignette("dataset", package = "arrow")`][25] for more.
+
+Another major area of work in this release was to improve the installation
+experience on Linux. A source package installation (as from CRAN) will now
+handle its C++ dependencies automatically, with no system dependencies beyond what R requires.
+For common Linux distributions and versions, installation will retrieve a prebuilt static
+C++ library for inclusion in the package; where this binary is not available,
+the package executes a bundled script that should build the Arrow C++ library.
+See [`vignette("install", package = "arrow")`][26] for details.
+
+For more on what's in the 0.16 R package, see the R [changelog][14].
+
+## Ruby and C GLib notes
+
+Ruby and C GLib continues to follow the features in the C++ project.
+
+### Ruby
+
+Ruby includes the following improvements.
+
+* Improve CSV save performance (ARROW-7474).
+* Add support for saving/loading TSV (ARROW-7454).
+* Add `Arrow::Schema#build_expression` to improve building `Gandiva::Expression` (ARROW-6619).
+
+### C GLib
+
+C GLib includes the following changes.
+
+* Add support for LargeList, LargeBinary, and LargeString (ARROW-6285, ARROW-6286).
+* Add filter and take API for GArrowTable, GArrowChunkedArray, and GArrowRecordBatch (ARROW-7110, ARROW-7111).
+* Add `garrow_table_combine_chunks()` API (ARROW-7369).
+
+## Rust notes
+
+Support for Arrow data types has been improved, with the following array types now supported (ARROW-3690):
+
+* Fixed Size List and Fixed Size Binary
+* Adding a String Array for utf-8 strings, and keeping the Binary Array for general binary data
+* Duration and interval arrays.
+
+Initial work on Arrow IPC support has been completed, with readers and writers for streams and files implemented (ARROW-5180).
+
+### Rust: DataFusion
+
+Query execution has been reimplemented with an extensible physical query plan. This allows other projects to add other plans, such as for distributed computing or for specific database servers (ARROW-5227).
+
+Added support for writing query results to CSV (ARROW-6274).
+
+The new Parquet -> Arrow reader is now used to read Parquet files (ARROW-6700).
+
+Various other query improvements have been implemented, especially on grouping and aggregate queries (ARROW-6689).
+
+### Rust: Parquet
+
+The Arrow reader integration has been completed, allowing Parquet files to be
+read into Arrow memory (ARROW-4059).
+
+## Development notes
+
+Arrow has moved away from Travis-CI and is now using Github Actions for
+PR-based continuous integration. This new CI configuration relies heavily
+on `docker-compose`, making it easier for developers to reproduce builds
+locally, thanks to tremendous work by Krisztián Szűcs (ARROW-7101).
+
+## Community Discussions Ongoing
+
+There are a number of active discussions ongoing on the developer
+dev@arrow.apache.org mailing list. We look forward to hearing from the
+community there.
+
+* Mandatory fields in IPC format: to ease input validation, it is being
+ [proposed][20] to mark some fields in our Flatbuffers schema "required".
+ Those fields are already semantically required, but are not considered so by
+ the generated Flatbuffers verifier. Before accepting this proposal, we need
+ to ensure that it does not break binary compatibility with existing valid
+ data.
+* The C Data Interface has not yet been formally adopted, though the community
+ has reached consensus to move forward after addressing various design
+ questions and concerns.
+* Guidelines for the use of "unsafe" in the Rust implementation are being
+ [discussed][22].
+
+[1]: https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.0
+[2]: https://arrow.apache.org/release/0.16.0.html#contributors
+[3]: https://arrow.apache.org/release/0.16.0.html
+[4]: https://github.com/eerhardt
+[5]: https://github.com/jorisvandenbossche
+[6]: https://google.github.io/oss-fuzz/
+[10]: https://github.com/apache/parquet-format/blob/master/Encryption.md
+[14]: https://arrow.apache.org/docs/r/news/
+[20]: https://mail-archives.apache.org/mod_mbox/arrow-dev/202001.mbox/%3C0dd13489-9221-459a-3560-1426738d3bb4%40python.org%3E
+[21]: https://mail-archives.apache.org/mod_mbox/arrow-dev/202001.mbox/%3CCAKa9qDkrcyL2UH0q1igDRwHeLWoE%2B%2B8Cac_%3DxbX%2Badoy-a0pSw%40mail.gmail.com%3E
+[22]: https://mail-archives.apache.org/mod_mbox/arrow-dev/202001.mbox/%3cMN2PR19MB347050FB22046485C0B2525EDD380@MN2PR19MB3470.namprd19.prod.outlook.com%3e
+[23]: https://arrow.apache.org/docs/format/Versioning.html
+[24]: https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/
+[25]: https://arrow.apache.org/docs/r/articles/dataset.html
+[26]: https://arrow.apache.org/docs/r/articles/install.html
+[27]: https://github.com/apache/arrow/commit/0ddc1f4737c35008cd06be1ee28472ebd7da68e2
+[28]: https://github.com/apache/arrow/blob/master/cpp/examples/minimal_build/build.sh
