| commit | 8a7007f03397849b555a74297a6a637293958cc5 | [log] [tgz] |
|---|---|---|
| author | Paul Rogers <par0328@yahoo.com> | Thu May 30 18:43:09 2019 -0700 |
| committer | Arina Ielchiieva <arina.yelchiyeva@gmail.com> | Fri Jun 07 13:25:20 2019 +0300 |
| tree | d7d9603a86fbf1aa02c025c63bbfc224fa97bd67 | |
| parent | 2615d68de4e44b1f03f5c047018548c06a7396b4 [diff] |
DRILL-7258: Remove field width limit for text reader The V2 text reader enforced a limit of 64K characters when using column headers, but not when using the columns[] array. The V3 reader enforced the 64K limit in both cases. This patch removes the limit in both cases. The limit now is the 16MB vector size limit. With headers, no one column can exceed 16MB. With the columns[] array, no one row can exceed 16MB. (The 16MB limit is set by the Netty memory allocator.) Added an "appendBytes()" method to the scalar column writer which adds additional bytes to those already written for a specific column or array element value. The method is implemented for VarChar, Var16Char and VarBinary vectors. It throws an exception for all other types. When used with a type conversion shim, the appendBytes() method throws an exception. This should be OK because, the previous setBytes() should have failed because a huge value is not acceptable for numeric or date types conversions. Added unit tests of the append feature, and for the append feature in the batch overflow case (when appending bytes causes the vector or batch to overflow.) Also added tests to verify the lack of column width limit with the text reader, both with and without headers. closes #1802
Apache Drill is a distributed MPP query layer that supports SQL and alternative query languages against NoSQL and Hadoop data storage systems. It was inspired in part by Google's Dremel.
Please read Environment.md for setting up and running Apache Drill. For complete developer documentation see DevDocs.md
Please see the Apache Drill Website or the Apache Drill Documentation for more information including:
Apache Drill is an Apache Foundation project and is seeking all types of users and contributions. Please say hello on the Apache Drill mailing list.You can also join our Google Hangouts or join our Slack Channel if you need help with using or developing Apache Drill. (More information can be found on Apache Drill website).
This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.
The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code. The following provides more details on the included cryptographic software: Java SE Security packages are used to provide support for authentication, authorization and secure sockets communication. The Jetty Web Server is used to provide communication via HTTPS. The Cyrus SASL libraries, Kerberos Libraries and OpenSSL Libraries are used to provide SASL based authentication and SSL communication.