Optiq release history

For a full list of releases, see github.

0.5 / 2014-03-14

New features

  • Allow quoting, quotedCasing, unquotedCasing, and caseSensitive properties to be specified explicitly (Vladimir Sitnikov)
  • Recognize more kinds of materializations, including filter-on-project (where project contains expressions) and some kinds of aggregation.
  • Fix #128, “Support WITH queries (common table expressions)”.
  • Fix #53, “Allow WHEN clause in simple CASE expression to have multiple values”.
  • Fix #156, “Optiq should recognize ‘SYSTEM TABLE’, ‘JOIN’, ‘INDEX’ as table types”.
  • Support querying ARRAY columns from JDBC source. (Gabriel Reid)

API changes

  • Add ProjectRelBase.copy(RelTraitSet, RelNode, List<RexNode>, RelDataType) and make ProjectRelBase.copy(RelTraitSet, RelNode) final. (This is a breaking change for sub-classes of ProjectRelBase.)
  • Change RexBuilder.makeRangeReference parameter type.
  • RexBuilder.makeInputRef replaces RelOptUtil.createInputRef.
  • Fix #160, “Allow comments in schema definitions”.
  • Fix #147, “Create a new kind of SqlCall that keeps operands in fields, not an operands array”. Very widely used parse tree nodes with complex operands, including SqlSelect, SqlJoin, SqlInsert, and a new node type SqlOrderBy, are now sub-classes of SqlCall but not SqlBasicCall. (This is a breaking change to code that assumes that, say, SqlSelect has an operands field.)
  • Convert all enum constants to upper-case. (This is a breaking change.)

Bug-fixes and internal changes

  • Generate optiq-core-VERSION-tests.jar not parent-VERSION-tests.jar.
  • Fix #176, “ORDER BY expression doesn't work with SELECT *”.
  • Fix VARCHAR casts sent to hsqldb source (Bruno Dumon)
  • Fix #143, “Remove dependency on eigenbase-resgen”.
  • Fix #173, “Case-insensitive table names are not supported for Casing.UNCHANGED”.
  • DATE.getLimit now returns Calendar in GMT time zone (Vladimir Sitnikov)
  • Set en_US locale in tests that match against error numbers, dates (Vladimir Sitnikov)
  • Use 1 test thread per CPU to avoid thread starvation on dual core CPUs (Vladimir Sitnikov)
  • Fix #174, “Move hsqldb to test scope”.
  • Add unit tests for RexExecutorImpl.
  • Correct JSON model examples in Javadoc comments. (Karel Vervaeke)
  • Move test reference logs from src/test/java to src/test/resources (reduces the number of ‘untracked files’ reported by git)
  • Tune Util.SpaceList, fix race condition, and move into new utility class Spaces.
  • Fix #163, “Equi-join warning”.
  • Fix #157, “Handle SQLFeatureNotSupported when calling setQueryTimeout”. (Karel Vervaeke)
  • Fix Optiq on Windows. (All tests and checkstyle checks pass.)
  • In checkstyle, support Windows-style file separator, otherwise build fails in Windows due to suppressions not used. (Vladimir Sitnikov)
  • Enable MongoDB tests when -Doptiq.test.mongodb=true.
  • Cleanup cache exception-handling and an assert.
  • Fix #153, “Error using MongoDB adapter: Failed to set setXIncludeAware(true)”.
  • Disable spark engine unless Spark libraries are on the class path and spark=true is specified in the connect string.
  • Fix path to mongo-zips-model.json in HOWTO. (Mariano Luna)
  • Fix bug deriving the type of a join-key.
  • Fix the value of ONE_MINUS_EPSILON.
  • Fix #158, “Optiq fails when call Planner.transform() multiple times, each with different ruleset”.
  • Fix #148, “Less verbose description of collation”. Also, optimize RelTraitSet creation and amortize RelTraitSet.toString().
  • Add generics to SQL parser.
  • Fix #145, “Unexpected upper-casing of keywords when using java lexer”.
  • Remove duplicate maven-source-plugin.
  • Fix #141, “Downgrade to guava-11.0.2”. This is necessary for Hadoop compatibility. Later versions of Guava can also be used.
  • Upgrade to spark-0.9.0. (Because this version of spark is available from maven-central, we can make optiq-spark part of the regular build, and remove the spark profile.)

0.4.18 / 2014-02-14

API and functionality changes

  • Configurable lexical policy
    • Fix #33, “SQL parser should allow different identifier quoting”.
    • Fix #34, “Policy for case-sensitivity of identifiers should be configurable”.
    • New connect-string parameter “lex”, with allowable values “ORACLE”, “MYSQL”, “SQL_SERVER”, “JAVA” sets policy to be like those databases, in terms of quote string, whether quoted and unquoted identifiers are converted to upper/lower case, and whether identifiers are matched case-sensitively. “JAVA” is case-sensitive, even for unquoted identifiers. It should be possible for each connection to have its own settings for these. Objects shared between sessions (views, materialized views) might require more work.
    • Added various internals to make it easy for developers to do the right thing. When you need to look up a schema, table or column/field name, you should use a catalog reader, and it will apply the right case-sensitivity policy.
    • Enable optiq consumer to utilize different lexical settings in Frameworks/Planner. (Jacques Nadeau)
  • Fix #115, “Add a PARSE_TREE hook point with SqlNode parameter”.
  • Change planner rules to use ProjectFactory for creating projects. (John Pullokkaran)
  • Fix #131, “Add interfaces for metadata (statistics)”. (This is a breaking change.)
  • Update Avatica to allow Cursor & Accessor implementations to throw SQLException. (Jacques Nadeau)
  • Separate cost model (RelOptCostFactory) from planner. Allow VolcanoPlanner to be sub-classed with different cost factory.
    • Remove references to VolcanoCost from RelSubset, so clients can use a different RelOptCost. (Harish Butani)
    • Make VolcanoCost immutable.
  • Break SqlTypeStrategies into OperandTypes, ReturnTypes and InferTypes, and rename its static members to upper-case, per checkstyle. (This is a breaking change.)
  • Add a mechanism for defining configuration parameters and have them appear in the responses to AvaticaDatabaseMetaData methods.
  • Fix #113, “User-defined scalar functions”.
  • Add rules to short-cut a query if LIMIT 0 is present. Also remove sort, aggregation, join if their inputs are known to be empty, and propagate the fact that the relational expressions are known to be empty up the tree. (We already do this for union, filter, project.)
  • RexNode and its sub-classes are now immutable.

Bug fixes and internal changes

  • Fix #16, “Upgrade to janino-2.7”.
  • Upgrade to guava-15.0 (guava-14.0.1 still allowed), sqlline-1.1.7, maven-surefire-plugin-2.16, linq4j-0.1.13.
  • Fix #136, “Support Hive dialect”.
  • Fix #138, “SqlDataTypeSpec.clone handles collection types wrong”.
  • Fix #137, “If a subset is created that is subsumed by an existing subset, its “best” is not assigned”.
    • If best rel in a Volcano subset doesn't have metadata, see if other rels have metadata.
  • Fix #127, “EnumerableCalcRel can't support 3+ AND conditions”. (Harish Butani)
  • Fix push-down of datetime literals to JDBC data sources.
  • Add Util.startsWith(List, List) and Util.hashCode(double).
  • Add maven-checkstyle-plugin, enable in “verify” phase, and fix exceptions.
  • Fix SqlValidator to rely on RelDataType to do field name matching. Fix RelDataTypeImpl to correctly use the case sensitive flag rather than ignoring it.
  • Fix #119, “Comparing Java type long with SQL type INTEGER gives wrong answer”.
  • Enable multi-threaded testing, and fix race conditions.
    • Two of the race conditions involved involving trait caches. The other was indeterminacy in type system when precision was not specified but had a default; now we canonize TIME to TIME(0), for instance.
  • Convert files to us-ascii.
  • Work around JANINO-169.
  • Refactor SQL validator testing infrastructure so SQL parser is configurable.
  • Add optiq-mat-plugin to README.
  • Fix the check for duplicate subsets in a rule match.
  • Fix #112, “Java boolean column should be treated as SQL boolean”.
  • Fix escaped unicode characters above 0x8000. Add tests for unicode strings.

0.4.17 / 2014-01-13

API changes

  • Fix #106, “Make Schema and Table SPIs simpler to implement, and make them re-usable across connections”. (This is a breaking change.)
  • Make it easier to define sub-classes of rule operands. The new class RelOptRuleOperandChildren contains the children of an operand and the policy for dealing with them. Existing rules now use the new methods to construct operands: operand(), leaf(), any(), none(), unordered(). The previous methods are now deprecated and will be removed before 0.4.18. (This is a breaking change.)
  • Fix #101, “Enable phased access to the Optiq engine”.
  • List-handling methods in Util: add methods skipLast, last, skip; remove subList, butLast.
  • Convert SqlIdentifier.names from String[] to ImmutableList<String>.
  • Rename OptiqAssert.assertThat() to that(), to avoid clash with junit's Assert.assertThat().
  • Usability improvements for RelDataTypeFactory.FieldInfoBuilder. It now has a type-factory, so you can just call build().
  • Rework HepProgramBuilder into a fluent API.
  • Fix #105, “Externalize RelNode to and from JSON”.

Tuning

  • If EnumerableAggregateRel has no aggregate functions, generate a call to Enumerable.distinct(), thereby saving the effort of building trivial accumulators.
  • Default rule set now does not introduce CalcRel until a later phase of planning. This reduces the number of trivial projects and calcs created, merged, and elimated.
  • Reduce the amount of time spent creating record types that already exist.
  • More efficient implementation of Util.isDistinct for small lists.
  • When an internal record has 0 fields, rather than generating a synthetic class and lots of instances that are all the same, use the new Unit class, which is a singleton.
  • To take advantage of asymmetric hash join added recently in linq4j, tweak cost of EnumerableJoinRel so that join is cheaper if the larger input is on the left, and more expensive if it is a cartesian product.
  • Fix #70, “Joins seem to be very expensive in memory”.
  • Make planning process more efficient by not sorting the list of matched rules each cycle. It is sorted if tracing is enabled; otherwise we scan to find the most important element. For this list, replace LinkedList with ChunkList, which has an O(1) remove and add, a fast O(n) get, and fast scan.

Other

  • Fix #87, “Constant folding”. Rules for constant-expression reduction, and to simplify/eliminate VALUES operator.
  • Graph algorithms: Implement breadth-first iterator and cycle-detector.
  • Fix bug in planner which occurred when two RelNodes have identical digest but different row-type.
  • Fix link to optiq-csv tutorial.
  • Fix bugs in RemoveTrivialProjectRule.strip, JdbcProjectRel.implement and SortRel.computeSelfCost.
  • Reformat code, and remove @author tags.
  • Upgrade to eigenbase-xom-1.3.4, eigenbase-properties-1.1.4, eigenbase-resgen-1.3.6.
  • Upgrade to linq4j-0.1.12.
  • Fix #97, correlated EXISTS.
  • Fix a bug in VolcanoCost.
  • Add class FoodMartQuerySet, that contains the 6,700 foodmart queries.
  • Fix factory class names in UnregisteredDriver
  • Fix #96, “LIMIT against a table in a clone schema causes UnsupportedOperationException”.
  • Disable spark module by default.
  • Allow CloneSchema to be specified in terms of url, driver, user, password; not just dataSource.
  • Wrap internal error in SQLException.

0.4.16 / 2013-11-24

  • Fix #69, “Can't join on string columns” and other problems with expressions in the join condition
  • Fix #74, “JOIN ... USING fails in 3-way join with UnsupportedOperationException”.
  • Fix #65 issues in the JDBC driver, and in particular to DatabaseMetaData methods, to make Squirrel-SQL run better. Fix JDBC column, table, schema names for when the table is not in a schema of depth 1.
  • Fix #85, “Adding a table to the root schema causes breakage in OptiqPrepareImpl”.
  • Extract Optiq‘s JDBC driver as a new JDBC driver framework, Avatica (#84). Other projects can use this to implement a JDBC driver by implementing just a few methods. If you wish to use Optiq’s JDBC driver, you will now need to include optiq-avatica.jar in addition to optiq-core.jar. Avatica does not depend on anything besides the standard Java library.
  • Support for parameters in PreparedStatement.
  • First steps in recognizing complex materializations. Internally we introduce a concept called a “star table”, virtual table composed of real tables joined together via many-to-one relationships. The queries that define materializations and end-user queries are canonized in terms of star tables. Matching (not done yet) will then be a matter of looking for sort, groupBy, project. It is not yet possible to define a star in an Optiq model file.
  • Add section to HOWTO on implementing adapters.
  • Fix data type conversions when creating a clone table in memory.
  • Fix how strings are escaped in JsonBuilder.
  • Test suite now depends on an embedded hsqldb database, so you can run mvn test right after pulling from git. You can instead use a MySQL database if you specify ‘-Doptiq.test.db=mysql’, but you need to manually populate it.
  • Fix a planner issue which occurs when the left and right children of join are the same relational expression, caused by a self-join query.
  • Fix #76 precedence of the item operator, map[index]; remove the space before ‘[’ when converting parse tree to string.
  • Allow CAST(expression AS ANY), and fix an issue with the ANY type and NULL values.
  • Handle null timestamps and dates coming out of JDBC adapter.
  • Add jdbcDriver attribute to JDBC schema in model, for drivers that do not auto-register.
  • Allow join rules to match any subclass of JoinRelBase.
  • Push projects, filters and sorts down to MongoDB. (Fixes #57, #60 and #72.)
  • Add instructions for loading FoodMart data set into MongoDB, and how to enable tracing.
  • Now runs on JDK 1.8 (still runs on JDK 1.6 and JDK 1.7).
  • Upgrade to junit-4.11 (avoiding the dodgy junit-4.1.12).
  • Upgrade to linq4j-0.1.11.

0.4.15 / 2013-10-14

  • Lots of good stuff that this margin is too small to contain. See a SQL language reference.