| <!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>PutBigQuery</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">PutBigQuery</h1><h2>Description: </h2><p>Unified processor for batch and stream flow files content to a Google BigQuery table via the Storage Write API.The processor is record based so the used schema is driven by the RecordReader. Attributes that are not matched to the target schemaare skipped. Exactly once delivery semantics are achieved via stream offsets. The Storage Write API is more efficient than the older insertAll method because it uses gRPC streaming rather than REST over HTTP</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>google, google cloud, bq, bigquery</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Display Name</th><th>API Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td>GCP Credentials Provider Service</td><td></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation: </strong><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td>gcp-project-id</td><td></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name"><strong>Dataset</strong></td><td>bq.dataset</td><td id="default-value">${bq.dataset}</td><td id="allowable-values"></td><td id="description">BigQuery dataset name (Note - The dataset must exist in GCP)<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Table Name</strong></td><td>bq.table.name</td><td id="default-value">${bq.table.name}</td><td id="allowable-values"></td><td id="description">BigQuery table name<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Record Reader</strong></td><td>bq.record.reader</td><td></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordReaderFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.grok.GrokReader/index.html">GrokReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.xml.XMLReader/index.html">XMLReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.avro.AvroReader/index.html">AvroReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.syslog.Syslog5424Reader/index.html">Syslog5424Reader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.cef.CEFReader/index.html">CEFReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.syslog.SyslogReader/index.html">SyslogReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.json.JsonTreeReader/index.html">JsonTreeReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.csv.CSVReader/index.html">CSVReader</a><br/><a href="../../../nifi-scripting-nar/1.19.1/org.apache.nifi.record.script.ScriptedReader/index.html">ScriptedReader</a><br/><a href="../../../nifi-parquet-nar/1.19.1/org.apache.nifi.parquet.ParquetReader/index.html">ParquetReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.json.JsonPathReader/index.html">JsonPathReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.lookup.ReaderLookup/index.html">ReaderLookup</a><br/><a href="../../../nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.windowsevent.WindowsEventLogReader/index.html">WindowsEventLogReader</a></td><td id="description">Specifies the Controller Service to use for parsing incoming data.</td></tr><tr><td id="name"><strong>Transfer Type</strong></td><td>bq.transfer.type</td><td id="default-value">STREAM</td><td id="allowable-values"><ul><li>STREAM <img src="../../../../../html/images/iconInfo.png" alt="Use streaming record handling strategy" title="Use streaming record handling strategy"></img></li><li>BATCH <img src="../../../../../html/images/iconInfo.png" alt="Use batching record handling strategy" title="Use batching record handling strategy"></img></li></ul></td><td id="description">Defines the preferred transfer type streaming or batching</td></tr><tr><td id="name"><strong>Append Record Count</strong></td><td>bq.append.record.count</td><td id="default-value">20</td><td id="allowable-values"></td><td id="description">The number of records to be appended to the write stream at once. Applicable for both batch and stream types</td></tr><tr><td id="name"><strong>Number of retries</strong></td><td>gcp-retry-count</td><td id="default-value">6</td><td id="allowable-values"></td><td id="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Skip Invalid Rows</strong></td><td>bq.skip.invalid.rows</td><td id="default-value">false</td><td id="allowable-values"></td><td id="description">Sets whether to insert all valid rows of a request, even if invalid rows exist. If not set the entire insert request will fail if it contains an invalid row.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google BigQuery operation.</td></tr><tr><td>failure</td><td>FlowFiles are routed to this relationship if the Google BigQuery operation fails.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>bq.records.count</td><td>Number of records successfully inserted</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch/index.html">PutBigQueryBatch</a>, <a href="../org.apache.nifi.processors.gcp.bigquery.PutBigQueryStreaming/index.html">PutBigQueryStreaming</a></p></body></html> |