This page provides frequently-asked questions about the Trafodion project and their answers.
Project Trafodion is an open source initiative originally cultivated by HP Labs and HP IT to develop an enterprise-class SQL-on-HBase solution targeted for big data transactional or operational workloads as opposed to analytic workloads.
Trafodion is currently available at downloads.trafodion.org. There, you can find an installer and executable code. For installation instructions, see Installation.
The key features are:
Trafodion delivers a full-featured and optimized transactional SQL-on-HBase DBMS solution with full transactional data protection. These capabilities help overcome basic Hadoop limitations in supporting transactional workloads.
With Trafodion, users gain the following benefits:
And operational SQL users also gain the following benefits inherent in Hadoop ecosystem.
Trafodion is Welsh for transactions.
Primary use cases on Trafodion consist of existing HBase workloads enhanced through SQL with transactional data protection. Another class of use cases consist of new or rehosted existing operational applications to address scalability issues, complex application programming, or prohibitive licensing costs.
Trafodion is based on HP’s 20+ year history in developing database products for OLTP. Building on this heritage, Trafodion provides distributed transaction management protection over multiple SQL statements, multiple tables and multiple rows.
Transactional or Operational SQL describe workloads were previously described simply as OLTP (online transaction processing) workloads. This expands that definition from the broad range of enterprise-level transactional applications (ERP, CRM, etc.) to include the new transactions generated from social and mobile data interactions and observations and the new mixing of structured and semi-structured data.
Trafodion is an open source product and like other products in the Hadoop ecosystem, it obtains two key advantages over the tradional proprietary OLTP database systems: Cost structure and Data reuse.
Not quite. Trafodion runs on top of HBase using the HBase APIs to perform the typical SQL functions of insert, update, delete, and query data with transactional protection. However, there are plans to include database capabilities using the coprocessor mechanism provided by HBase.
With Trafodion, users can make use of standard HBase APIs as well as use Trafodion for SQL access to the HBase tables. This includes Trafodion's ability to use powerful SQL capabilities to do joins against both HBase and Trafodion tables and run queries against them while providing transaction management capability.
This allows application developers to choose the best access methods for each particular data usage, based on skill set, data size and usage, and access pattern requirements.
Yes, Trafodion provides full-featured ANSI DML operations including INSERT, UPDATE, DELETE, MERGE, and UPSERT statement support.
Benchmark results will be posted as the community publishes them.
Trafodion is hosted on top of HBase and HDFS. HBase is generally viewed as being CA (Consistent, Available) in the context of the CAP theorem.
Unlike regular HBase, Trafodion extends the definition of consistency to provide ACID protection across transactions comprised of multiple SQL statements, tables, and rows.
As Trafodion is hosted on HBase/HDFS, in theory, Trafodion’s database should scale as HBase/HDFS scales. To this point (due to available hardware limitations), Trafodion has been tested with configurations up to 10 nodes and 50 terabytes.
HBase and HDFS are considered highly available with many built in features for HA including name node redundancy, HDFS k-safety data replication, HBase replication, HBase snapshots, Zookeeper’s highly reliable distributed coordination of Hadoop hosted services, and so on.
Additionally, the planned release of HBase 1.0 is advertised as having many additional HA features which Trafodion incorporates as it supports HBase 1.0. Furthermore, many Hadoop distributions have added their own HA features at the HBase or HDFS layers.
Trafodion leverages HBase and HDFS capabilities for providing extended HA to the connectivity and SQL layers. For example, Trafodion registers its connection services using Zookeeper to ensure persistent connectivity services. Trafodion is designed as an MPP service with replicated service points for HA and no single points of failure.
Trafodion provides a bulk load capability that interfaces directly with HDFS for high-performance data loading. HBase leverages large-scale cached memory to host data in memory until the HFile buffer becomes filled and is then written to HDFS.
Additionally, Trafodion provides many low-latency read optimizations on top of HBase to significantly improve the both the random and parallel read performance of the database engine. These include:
Trafodion provides SQL search capabilities using search functions such as POSITION (equivalent to INDEX) and SUBSTRING. Once large object (LOB) support is made available, a user-written or third-party UDF could be plugged in to add search capabilities into documents and images. This is a great area for community contribution.
No, Trafodion supports full ACID properties. Distributed Transaction Management provides transaction consistency across multiple row updates, updates across multiple tables, and transactions spanning multiple update statements. At the return from a COMMIT WORK statement, all rows and tables are in consistent form with regards to that transaction.
Yes, Trafodion supports transaction management for native HBase tables.
Yes, all the reads in Trafodion are completely consistent.
Yes, Trafodion offers full ACID protection across multiple SQL statements, tables, or rows. Transactions can be aborted using the ROLLBACK WORK statement.
Not really. There are two logs that the transaction manager uses to coordinate transaction history, one at an HBase regionserver level and the other at the transaction manager level. But there is no support for shipping and replaying these logs on a remote DR site in a transactionally consistent manner.
There is no support for transaction management between Trafodion instances in different data centers.
Trafodion employs a Distributed Transaction Manager (DTM) model where a DTM runs on every datanode and each DTM works with its peers in other datanodes to coordinate the two-phase commit protocol. For efficiency purposes, the DTM is invoked only when necessary:
Trafodion has not been tested with Yarn yet, but there is nothing in Trafodion that precludes it from being treated as a regular application running in a Yarn container.
Trafodion does provide the capability for SQL statements submitted to Trafodion to access, join, and aggregate a combination of Trafodion database objects, native HBase objects, and Hive tables (via the HCatalog).
While theoretically possible since Trafodion data is stored in HBase/HDFS, the data would not be easily interpreted due to the encoding mechanisms Trafodion employs for transaction performance and efficiency.
Trafodion provides automatic access parallelism without the need for writing MapReduce. If the optimized plan calls for parallel execution, the Trafodion Master process divides the work among Executive Server Processes (ESPs) to perform the work in parallel on behalf of the Master process. The results are passed back to the Master for consolidation.
In some situations where there a highly complex plan specified (for example, large n-way joins or aggregations), multiple layers of ESPs may be requested. See the diagram below.
While theoretically possible since Trafodion data is stored in HBase/HDFS, the data would not be easily interpreted due to the encoding mechanisms Trafodion employs for transaction performance and efficiency.
Trafodion maintains its own catalog for metadata that provides the relational abstraction layer for accessing the physical storage layer via SQL commands. Trafodion objects are complex as they can have tables, indexes, views, constraints, which are represented in Trafodion’s own catalog. HCatalog can represent only simple objects like tables or column families currently but does not contain support for representing other more complex relational objects.
Alternatively, Trafodion data could be loaded into Hive/HDFS and then accessed using Hive specific operations or native MapReduce jobs.
Currently, Trafodion provides security management based on HBase capabilities. There are projects as part of roadmap to provide full GRANT/REVOKE capability at the table and schema level for users and roles, along with support for directory based authentication.
Yes, Trafodion supports ANSI GRANT/REVOKE on both an individual user and role basis. See Enabling Security Features in Trafodion. For details on the GRANT/REVOKE syntax, see the Trafodion SQL Reference Manual (pdf, 3.98 MB).
Trafodion supports authorization (that is, privileges) at the system and database-object level. System privileges are valid across SQL, such as the ability to create schemas and objects. Object privileges include tables, views, and other databse objects. Views can be used to restrict access to the subset of table rows referenced within the view.
Trafodion is designed to be hosted on vendor neutral, commodity hardware. As Trafodion runs on top of HBase and HDFS, a vendor's recommended reference architectur such as HP Reference Architecture for Hadoop or similar server configuration is recommended as a starting point for a new installation.
Trafodion needs to be installed on all nodes that host an HBase RegionServer. Trafodion comes with an install script to make the installation process easy. For more information, see the Trafodion installation instructions.
Trafodion software requires an x86-64 bit hardware platform running RedHat Enterprise Linux or CentOS 6.x kernel (64-bit). It supports Cloudera and Hortonworks Hadoop distributions; for details, see Supported Hadoop Distributions. The software is intended to be Linux and Hadoop distribution neutral, but other distributions have not been tested at this time.
Recommended sizings for Trafodion are still being developed, and vary based on the amount of data, anticipated query workload (types, rates/respose times, concurrency) and the peak number of supported concurrent connections).
What’s the process like when we add a new node to the cluster? Does the system grow elastically? The underlying HBase/HDFS infrastructure supports elastic scalability allowing for incremental node expansion, that is, datanode expansion. Trafodion could then access the HBase tables (or ‘regions’ of the HBase tables) that now span the new node(s). Trafodion services (that is, connection, compiler, master, ESP) can be made to leverage these additional nodes by restarting Trafodion with a new coniguration.
Trafodion can run on dedicated hardware, cloud or VM configurations. For best and consistent performance, dedicated hardware is recommended where feasible.