| <!DOCTYPE html> |
| <html class="no-js" lang="en"> |
| <head> |
| <title>Apache NiFi MiNiFi Getting Started</title> |
| <meta charset="utf-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <link rel="shortcut icon" href="/assets/images/nifi16.ico"/> |
| <link rel="stylesheet" href="/stylesheets/app.min.27f785d2dfa83c24896261ad5106f3c19ee004c4d04b8a1c4b22e03a3a9fcb2a.css"> |
| <link rel="stylesheet" href="/assets/stylesheets/font-awesome.min.css"> |
| <script src="/assets/js/modernizr.js"></script> |
| <script src="/assets/js/webfontloader.js"></script> |
| <script> |
| var _paq = window._paq = window._paq || []; |
| _paq.push(['disableCookies']); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var baseUri = "//matomo.privacy.apache.org/"; |
| _paq.push(['setTrackerUrl', baseUri + 'matomo.php']); |
| _paq.push(['setSiteId', '28']); |
| var trackerElement = document.createElement('script'); |
| var firstScriptElement = document.getElementsByTagName('script')[0]; |
| trackerElement.async = true; |
| trackerElement.src = baseUri + 'matomo.js'; |
| var firstScriptElement = document.getElementsByTagName('script')[0]; |
| if (window.location.host === 'nifi.apache.org') { |
| firstScriptElement.parentNode.insertBefore(trackerElement, firstScriptElement); |
| } |
| })(); |
| </script> |
| </head> |
| <body><div class="sticky contain-to-grid"> |
| <nav class="top-bar" data-topbar role="navigation"> |
| <ul class="title-area"> |
| <li class="name"> |
| <h1> |
| <a href="index.html"> |
| <img id="logo-top-bar" src="/assets/images/minifi-drop-white.svg" alt="MiNiFi"/> |
| </a> |
| </h1> |
| </li> |
| |
| <li class="toggle-topbar menu-icon"><a href="#"><span></span></a></li> |
| </ul> |
| |
| <section class="top-bar-section"> |
| |
| <ul class="right"> |
| <li class="has-dropdown"> |
| <a href="#">Documentation</a> |
| <ul class="dropdown"> |
| <li><a href="getting-started.html"><i class="fa"></i>Getting Started</a></li> |
| <li><a href="minifi-java-agent-quick-start.html"><i class="fa"></i>MiNiFi Java Quick Start Guide</a></li> |
| <li><a href="system-admin-guide.html"><i class="fa"></i>Administrator's Guide</a></li> |
| <li><a href="minifi-toolkit.html"><i class="fa"></i>Converter Toolkit</a></li> |
| <li><a href="faq.html"><i class="fa"></i>FAQ</a></li> |
| <li><a href="https://cwiki.apache.org/confluence/display/MiNiFi"><i class="fa fa-external-link external-link"></i>Wiki</a></li> |
| </ul> |
| </li> |
| <li class="has-dropdown"> |
| <a href="#">Downloads</a> |
| <ul class="dropdown"> |
| <li><a href="download.html">Download MiNiFi Components</a></li> |
| <li><a href="https://cwiki.apache.org/confluence/display/MINIFI/Release+Notes"><i class="fa fa-external-link external-link"></i>MiNiFi Java Release Notes</a></li> |
| <li><a href="https://cwiki.apache.org/confluence/display/MINIFI/Release+Notes#ReleaseNotes-MiNiFi(C++)"><i class="fa fa-external-link external-link"></i>MiNiFi C++ Release Notes</a></li> |
| </ul> |
| </li> |
| <li class="has-dropdown"> |
| <a href="#">Community</a> |
| <ul class="dropdown"> |
| <li><a href="https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide"><i class="fa fa-external-link external-link"></i>Contributor Guide</a></li> |
| <li><a href="../mailing_lists.html">Mailing Lists & Chat</a></li> |
| <li><a href="../people.html">People</a></li> |
| </ul> |
| </li> |
| <li class="has-dropdown"> |
| <a href="#">Development</a> |
| <ul class="dropdown"> |
| <li><a href="../release-guide.html">Release Guide</a></li> |
| <li><a href="../licensing-guide.html">Licensing Guide</a></li> |
| <li><a href="https://gitbox.apache.org/repos/asf?p=nifi.git"><i class="fa fa-external-link external-link"></i>Source - Java Agent</a></li> |
| <li><a href="https://github.com/apache/nifi"><i class="fa fa-external-link external-link"></i>Source - GitHub - Java Agent</a></li> |
| <li><a href="https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git"><i class="fa fa-external-link external-link"></i>Source - C++ Agent</a></li> |
| <li><a href="https://github.com/apache/nifi-minifi-cpp"><i class="fa fa-external-link external-link"></i>Source - GitHub - C++ Agent</a></li> |
| <li><a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20%22MiNiFi%22%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC"><i class="fa fa-external-link external-link"></i>MiNiFi Java Issues</a></li> |
| <li><a href="https://issues.apache.org/jira/browse/MINIFICPP"><i class="fa fa-external-link external-link"></i>MiNiFi C++ Issues</a></li> |
| </ul> |
| </li> |
| <li class="has-dropdown"> |
| <a href="#">ASF Links</a> |
| <ul class="dropdown"> |
| <li><a href="https://www.apache.org"><i class="fa fa-external-link external-link"></i>Apache Software Foundation</a></li> |
| <li><a href="https://privacy.apache.org/policies/privacy-policy-public.html"><i class="fa fa-external-link external-link"></i>Privacy Policy</a></li> |
| <li><a href="https://www.apache.org/foundation/sponsorship.html"><i class="fa fa-external-link external-link"></i>Sponsorship</a></li> |
| <li><a href="https://www.apache.org/security/"><i class="fa fa-external-link external-link"></i>Security</a></li> |
| <li><a href="https://www.apache.org/foundation/thanks.html"><i class="fa fa-external-link external-link"></i>Thanks</a></li> |
| </ul> |
| </li> |
| <li> |
| <a href="../index.html">Apache NiFi</a> |
| </li> |
| </ul> |
| </section> |
| </nav> |
| </div> |
| <div class="row"> |
| <div class="large-12 columns"> |
| <div class="large-space"></div> |
| <h1 id="getting-started-with-minifi">Getting started with MiNiFi</h1> |
| <p>This page explains how to configure and deploy MiNiFi agents.</p> |
| <p>The Java agent is able to run most of <a href="http://nifi.apache.org/docs.html">NiFi’s available processors</a>, but is a larger binary distribution (49MB) and consumes greater system resources (24MB max JVM heapsize by default). If you need maximum flexibility to make routing and processing decisions at your data’s point of origin, the Java agent is a good fit.</p> |
| <p>The C++ agent is a smaller binary (3.2MB), consumes low system memory (about 5MB at idle) but has <a href="https://github.com/apache/nifi-minifi-cpp#caveats">a limited subset of processors</a>. If your primary concern is gathering and pushing data to downstream consumers and minimizing system impact, the C++ agent is a good fit.</p> |
| <ol> |
| <li>Install the appropriate OS level dependencies:</li> |
| </ol> |
| <h4 id="minifi-java">MiNiFi Java:</h4> |
| <ul> |
| <li>Java 1.8+</li> |
| </ul> |
| <h4 id="minifi-c">MiNiFi C++:</h4> |
| <h6 id="rhelcentos">RHEL/CentOS:</h6> |
| <ul> |
| <li>yum install -y epel-release</li> |
| <li>yum install -y leveldb</li> |
| </ul> |
| <h6 id="debianubuntu">Debian/Ubuntu:</h6> |
| <ul> |
| <li>apt install -y libleveldb-dev</li> |
| <li>apt install -y libxml2</li> |
| </ul> |
| <ol start="2"> |
| <li>Download the relevant compressed binary from the <a href="download.html">Downloads</a> page</li> |
| <li>Copy and decompress the binary to your target deployment environment</li> |
| <li>Set the MINIFI_HOME environment variable to your decompressed binary directory</li> |
| <li>From $MINIFI_HOME, ./bin/minifi.sh {start|stop|run|restart|status|flowStatus|dump|install}</li> |
| </ol> |
| <p><em>flowStatus</em> and <em>dump</em> are available only for MiNiFi Java.</p> |
| <p>For MiNiFi Java, the agent logs to $MINIFI_HOME/logs/minifi-app.log.</p> |
| <p>For MiNiFi C++, the agent logs to $MINIFI_HOME/minifi-app.log.txt</p> |
| <h1 id="configuring-minifi-agent-flows">Configuring MiNiFi Agent Flows</h1> |
| <p>Agents execute the flow defined in $MINIFI_HOME/conf/config.yml. The shipped config.yml file has an “empty” flow.</p> |
| <p>While it’s possible to create and modify config.yml programmatically or by hand, the more common scenario is to:</p> |
| <ol> |
| <li>Use NiFi’s web interface to define a flow</li> |
| <li>Save the flow as a template and download its XML file</li> |
| <li>Use the <a href="minifi-toolkit.html">Converter Toolkit</a> to convert the template XML to an equivalent config.yml</li> |
| <li>Put the config.yml file in $MINIFI_HOME/conf and start the agent.</li> |
| </ol> |
| <h1 id="pushing-data-from-minifi-to-nifi">Pushing Data From MiNiFi to NiFi</h1> |
| <p>What follows is an illustrated walkthrough of configuring MiNiFi and NiFi to generate and collect log data.</p> |
| <p>If you’re new to NiFi, watch the <a href="http://nifi.apache.org/videos.html">NiFi introductory video series</a> before proceeding. If you don’t already have NiFi, you can download it from <a href="http://nifi.apache.org/download.html">here</a>.</p> |
| <h2 id="configuring-nifi-to-receive-data">Configuring NiFi to Receive Data</h2> |
| <p>First, check that $NIFI_HOME/conf/nifi.properties has a Site to Site input socket port specified:</p> |
| <pre tabindex="0"><code># Site to Site properties |
| nifi.remote.input.host= |
| nifi.remote.input.secure=false |
| nifi.remote.input.socket.port=1026 |
| nifi.remote.input.http.enabled=true |
| nifi.remote.input.http.transaction.ttl=30 sec |
| </code></pre><p>Restart NiFi so changes take effect.</p> |
| <p>On NiFi’s web UI, place an Input Port “From MiNiFi” and connect it to a downstream processor: |
| <img src="../assets/images/minifi/nifi-receive.png" alt=""></p> |
| <h2 id="creating-the-minifi-flow">Creating the MiNiFi Flow</h2> |
| <p>Create a new process group, and double click on it to enter it. Working inside this new processor group will let us create a clean template that includes flow components meant to run in MiNiFi.</p> |
| <p>Inside the process group, create a Remote Process Group (RPG) and give it the URL of your NiFi instance. The RPG will connect to NiFi and discover the input port created earlier.</p> |
| <p>Create a TailFile processor and connect it to the RPG, selecting the “From MiNiFi” input port. |
| <img src="../assets/images/minifi/minifi-flow.png" alt=""></p> |
| <p>Now create a template, download it as an XML file, and download the <a href="download.html">Converter ToolKit</a>.</p> |
| <p>From the toolkit directory:</p> |
| <pre tabindex="0"><code>[user@host minifi-toolkit-0.0.1]# ./bin/config.sh transform /path/to/template.xml /output/path/config.yml |
| |
| Java home: /usr/lib/jvm/jre-1.8.0-openjdk |
| MiNiFi Toolkit home: /minifi-toolkit-0.0.1 |
| |
| |
| No validation errors found in template. |
| </code></pre><p>Open the config.yml file and add Port and Host to a Properties section at the end of the Input Ports section (at the end of the file):</p> |
| <pre tabindex="0"><code>Remote Processing Groups: |
| - name: http://localhost:8080/nifi |
| url: http://localhost:8080/nifi |
| comment: '' |
| timeout: 30 sec |
| yield period: 10 sec |
| Input Ports: |
| - id: AUTOGENERATED_NIFI_PORT_ID_HERE |
| name: MiNiFi-input |
| comment: '' |
| max concurrent tasks: 1 |
| use compression: false |
| Properties: # Deviates from spec and will later be removed when this is autonegotiated |
| Port: 1026 |
| Host Name: localhost |
| </code></pre><p>Copy the generated config.yml file into $MINIFI/conf/config.yml, and start MiNiFi:</p> |
| <pre tabindex="0"><code>[user@host nifi-minifi-cpp-0.1.0]# ./bin/minifi.sh start |
| Starting MiNiFi with PID 463 and pid file /nifi-minifi-cpp-0.1.0/bin/.minifi.pid |
| </code></pre><p>After a few seconds, you should see NiFi receiving data on the input port: |
| <img src="../assets/images/minifi/nifi-received.png" alt=""></p> |
| <p>If you do not see data in the queue, it may have already been consumed by the downstream processor (PutFile in this case). If no data is arriving on the input port at all, check the MiNiFi log files, and confirm that the file being tailed exists and is not empty.</p> |
| |
| </div> |
| </div> |
| <div class="row"> |
| <div class="large-12 columns footer"> |
| <a href="https://www.apache.org"> |
| <img id="asf-logo" alt="Apache Software Foundation" src="/assets/images/asf_logo.png" width="200" style="margin:0px 10px" /> |
| </a> |
| <a href="https://www.apache.org/events/current-event.html"> |
| <img src="https://www.apache.org/events/current-event-234x60.png" style="margin:0px 10px" /> |
| </a> |
| <div id="copyright"> |
| <p>Copyright © 2023 The Apache Software Foundation, Licensed under the <a |
| href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.<br/>Apache, the |
| Apache feather logo, NiFi, Apache NiFi and the project logo are trademarks of The Apache Software |
| Foundation.</p> |
| </div> |
| </div> |
| </div> |
| <script src="/assets/js/jquery.min.js"></script> |
| <script src="/assets/js/foundation.js"></script> |
| <script src="/assets/js/app.js"></script> |
| </body> |
| </html> |
