#Gearpump Kudu
Gearpump integration for Apache Kudu
The message type that KuduSink is able to handle including:
Suppose there is a DataSource Task will output above-mentioned messages, you can write a simple application then:
val sink = new KuduSink(UserConfig.empty, "$tableName") val sinkProcessor = DataSinkProcessor(sink, "$sinkNum") val split = Processor[DataSource]("$splitNum") val computation = split ~> sinkProcessor val application = StreamApplication("Kudu", Graph(computation), UserConfig.empty)
The Kudu cluster should run on where Gearpump is deployed. Suppose Kdudu is installed at /usr/lib/kudu
on every node and you already have your application built into a jar file. Please note only client side's configuration change is needed. After that, you are able to submit the application.
val map = Map[String, String]("KUDUSINK" -> "kudusink", "kudu.masters"->"kuduserver", "KUDU_USER" -> "kudu.user", "GEARPUMP_KERBEROS_PRINCIPAL" -> "gearpump.kerberos.principal", "GEARPUMP_KEYTAB_FILE" -> "gearpump.keytab.file", "TABLE_NAME" -> "kudu.table.name" )
Before running job make sure you run kinit, with just a kinit you should be able to run the job and insert records into Kudu table