| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| [<img src="https://nifi.apache.org/assets/images/apache-nifi-logo.svg" width="300" height="126" alt="Apache NiFi"/>][nifi] |
| |
| [](https://github.com/apache/nifi/actions/workflows/ci-workflow.yml) |
| [](https://github.com/apache/nifi/actions/workflows/system-tests.yml) |
| [](https://hub.docker.com/r/apache/nifi/) |
| [](https://nifi.apache.org/download.html) |
| [](https://s.apache.org/nifi-community-slack) |
| |
| [Apache NiFi](https://nifi.apache.org/) is an easy to use, powerful, and |
| reliable system to process and distribute data. |
| |
| ## Table of Contents |
| |
| - [Features](#features) |
| - [Requirements](#requirements) |
| - [Getting Started](#getting-started) |
| - [MiNiFi subproject](#minifi-subproject) |
| - [Registry subproject](#registry-subproject) |
| - [Getting Help](#getting-help) |
| - [Documentation](#documentation) |
| - [License](#license) |
| - [Export Control](#export-control) |
| |
| ## Features |
| |
| Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, and system mediation logic. Some of its key features include: |
| |
| - Web-based user interface |
| - Seamless experience for design, control, and monitoring |
| - Multi-tenant user experience |
| - Highly configurable |
| - Loss tolerant vs guaranteed delivery |
| - Low latency vs high throughput |
| - Dynamic prioritization |
| - Flows can be modified at runtime |
| - Back pressure |
| - Scales up to leverage full machine capability |
| - Scales out with zero-leader clustering model |
| - Data Provenance |
| - Track dataflow from beginning to end |
| - Designed for extension |
| - Build your own processors and more |
| - Enables rapid development and effective testing |
| - Secure |
| - SSL, SSH, HTTPS, encrypted content, etc... |
| - Pluggable fine-grained role-based authentication/authorization |
| - Multiple teams can manage and share specific portions of the flow |
| |
| ## Minimum Requirements |
| * JDK 8 Update 251 |
| * Apache Maven 3.6.0 |
| |
| ## Getting Started |
| |
| Read through the [quickstart guide for development](https://nifi.apache.org/quickstart.html). |
| It will include information on getting a local copy of the source, give pointers on issue |
| tracking, and provide some warnings about common problems with development environments. |
| |
| For a more comprehensive guide to development and information about contributing to the project |
| read through the [NiFi Developer's Guide](https://nifi.apache.org/developer-guide.html). |
| |
| ### Building |
| |
| Run `mvn clean install` or for parallel build execute `mvn -T 2.0C clean install`. |
| |
| The parallel build should take around fifteen minutes on modern hardware. |
| |
| laptop:nifi myuser$ mvn -T 2.0C clean install |
| [INFO] Scanning for projects... |
| [INFO] Inspecting build with total of 115 modules... |
| ...tens of thousands of lines elided... |
| [INFO] ------------------------------------------------------------------------ |
| [INFO] BUILD SUCCESS |
| [INFO] ------------------------------------------------------------------------ |
| [INFO] Total time: 09:24 min (Wall Clock) |
| [INFO] Finished at: 2015-04-30T00:30:36-05:00 |
| [INFO] Final Memory: 173M/1359M |
| [INFO] ------------------------------------------------------------------------ |
| |
| Run `mvn clean install -DskipTests` to skip unit test execution. |
| |
| ### Deploying |
| |
| Change directories to `nifi-assembly`. The `target` directory contains binary archives. |
| |
| laptop:nifi myuser$ cd nifi-assembly |
| laptop:nifi-assembly myuser$ ls -lhd target/nifi* |
| drwxr-xr-x 3 myuser mygroup 102B Apr 30 00:29 target/nifi-1.0.0-SNAPSHOT-bin |
| -rw-r--r-- 1 myuser mygroup 144M Apr 30 00:30 target/nifi-1.0.0-SNAPSHOT-bin.tar.gz |
| -rw-r--r-- 1 myuser mygroup 144M Apr 30 00:30 target/nifi-1.0.0-SNAPSHOT-bin.zip |
| |
| Copy the `nifi-VERSION-bin.tar.gz` or `nifi-VERSION-bin.zip` to a separate deployment directory. |
| Extracting the distribution will create a new directory named for the version. |
| |
| laptop:nifi-assembly myuser$ mkdir ~/example-nifi-deploy |
| laptop:nifi-assembly myuser$ tar xzf target/nifi-*-bin.tar.gz -C ~/example-nifi-deploy |
| laptop:nifi-assembly myuser$ ls -lh ~/example-nifi-deploy/ |
| total 0 |
| drwxr-xr-x 10 myuser mygroup 340B Apr 30 01:06 nifi-1.0.0-SNAPSHOT |
| |
| ### Starting |
| |
| Change directories to the deployment location and run the following command to start NiFi. |
| |
| laptop:~ myuser$ cd ~/example-nifi-deploy/nifi-* |
| laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh start |
| |
| Running `bin/nifi.sh start` starts NiFi in the background and exits. Use `--wait-for-init` with an optional timeout in |
| seconds to wait for a complete startup before exiting. |
| |
| laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh start --wait-for-init 120 |
| |
| ### Authenticating |
| |
| The default configuration generates a random username and password on startup. NiFi writes the generated credentials |
| to the application log located in `logs/nifi-app.log` under the NiFi installation directory. |
| |
| The following command can be used to find the generated credentials on operating systems with `grep` installed: |
| |
| laptop:nifi-1.0.0-SNAPSHOT myuser$ grep Generated logs/nifi-app*log |
| |
| NiFi logs the generated credentials as follows: |
| |
| Generated Username [USERNAME] |
| Generated Password [PASSWORD] |
| |
| The `USERNAME` will be a random UUID composed of 36 characters. The `PASSWORD` will be a random string composed of |
| 32 characters. The generated credentials will be stored in `conf/login-identity-providers.xml` with the password stored |
| using bcrypt hashing. Record these credentials in a secure location for access to NiFi. |
| |
| The random username and password can be replaced with custom credentials using the following command: |
| |
| ./bin/nifi.sh set-single-user-credentials <username> <password> |
| |
| ### Running |
| |
| Open the following link in a web browser to access NiFi: https://localhost:8443/nifi |
| |
| The web browser will display a warning message indicating a potential security risk due to the self-signed |
| certificate NiFi generated during initialization. Accepting the potential security risk and continuing to load |
| the interface is an option for initial development installations. Production deployments should provision a certificate |
| from a trusted certificate authority and update the NiFi keystore and truststore configuration. |
| |
| Accessing NiFi after accepting the self-signed certificate will display the login screen. |
|  |
| |
| Using the generated credentials, enter the generated username in the `User` field |
| and the generated password in the `Password` field, then select `LOG IN` to access the system. |
|  |
| |
| ### Configuring |
| |
| The [NiFi User Guide](https://nifi.apache.org/docs/nifi-docs/html/user-guide.html) describes how to build a data flow. |
| |
| ### Stopping |
| |
| Run the following command to stop NiFi: |
| |
| laptop:~ myuser$ cd ~/example-nifi-deploy/nifi-* |
| laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh stop |
| |
| ## MiNiFi subproject |
| |
| MiNiFi is a child project effort of Apache NiFi. It is a complementary data collection approach that supplements the core tenets of [NiFi](https://nifi.apache.org/) in dataflow management, focusing on the collection of data at the source of its creation. |
| |
| Specific goals for MiNiFi are comprised of: |
| - small and lightweight footprint |
| - central management of agents |
| - generation of data provenance |
| - integration with NiFi for follow-on dataflow management and full chain of custody of information |
| |
| Perspectives of the role of MiNiFi should be from the perspective of the agent acting immediately at, or directly adjacent to, source sensors, systems, or servers. |
| |
| To run: |
| - Change directory to 'minifi-assembly'. In the target directory, there should be a build of minifi. |
| |
| $ cd minifi-assembly |
| $ ls -lhd target/minifi* |
| drwxr-xr-x 3 user staff 102B Jul 6 13:07 minifi-1.14.0-SNAPSHOT-bin |
| -rw-r--r-- 1 user staff 39M Jul 6 13:07 minifi-1.14.0-SNAPSHOT-bin.tar.gz |
| -rw-r--r-- 1 user staff 39M Jul 6 13:07 minifi-1.14.0-SNAPSHOT-bin.zip |
| |
| - For testing ongoing development you could use the already unpacked build present in the directory |
| named "minifi-*version*-bin", where *version* is the current project version. To deploy in another |
| location make use of either the tarball or zipfile and unpack them wherever you like. The |
| distribution will be within a common parent directory named for the version. |
| |
| $ mkdir ~/example-minifi-deploy |
| $ tar xzf target/minifi-*-bin.tar.gz -C ~/example-minifi-deploy |
| $ ls -lh ~/example-minifi-deploy/ |
| total 0 |
| drwxr-xr-x 10 user staff 340B Jul 6 01:06 minifi-1.14.0-SNAPSHOT |
| |
| To run MiNiFi: |
| - Change directory to the location where you installed MiNiFi and run it. |
| |
| $ cd ~/example-minifi-deploy/minifi-* |
| $ ./bin/minifi.sh start |
| |
| - View the logs located in the logs folder |
| $ tail -F ~/example-minifi-deploy/logs/minifi-app.log |
| |
| - For help building your first data flow and sending data to a NiFi instance see the System Admin Guide located in the docs folder or making use of the minifi-toolkit, which aids in adapting NiFi templates to MiNiFi YAML configuration file format. |
| |
| - If you are testing ongoing development, you will likely want to stop your instance. |
| |
| $ cd ~/example-minifi-deploy/minifi-* |
| $ ./bin/minifi.sh stop |
| |
| ### Docker Build |
| |
| To build: |
| - Run a full NiFi build (see above for instructions). Then from the minifi/ subdirectory, execute `mvn -P docker clean install`. This will run the full build, create a docker image based on it, and run docker-compose integration tests. After it completes successfully, you should have an apacheminifi:${minifi.version} image that can be started with the following command (replacing ${minifi.version} with the current maven version of your branch): |
| ``` |
| docker run -d -v YOUR_CONFIG.YML:/opt/minifi/minifi-${minifi.version}/conf/config.yml apacheminifi:${minifi.version} |
| ``` |
| |
| ## Registry subproject |
| |
| Registry—a subproject of Apache NiFi—is a complementary application that provides a central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi. |
| |
| ### Getting Registry Started |
| |
| 1) Build NiFi (see [Getting Started for NiFi](#getting-started) ) |
| |
| or |
| |
| Build only the Registry subproject: |
| |
| cd nifi/nifi-registry |
| mvn clean install |
| |
| |
| If you wish to enable style and license checks, specify the contrib-check profile: |
| |
| mvn clean install -Pcontrib-check |
| |
| |
| 2) Start Registry |
| |
| cd nifi-registry/nifi-registry-assembly/target/nifi-registry-<VERSION>-bin/nifi-registry-<VERSION>/ |
| ./bin/nifi-registry.sh start |
| |
| Note that the application web server can take a while to load before it is accessible. |
| |
| 3) Accessing the application web UI |
| |
| With the default settings, the application UI will be available at [http://localhost:18080/nifi-registry](http://localhost:18080/nifi-registry) |
| |
| 4) Accessing the application REST API |
| |
| If you wish to test against the application REST API, you can access the REST API directly. With the default settings, the base URL of the REST API will be at `http://localhost:18080/nifi-registry-api`. A UI for testing the REST API will be available at [http://localhost:18080/nifi-registry-api/swagger/ui.html](http://localhost:18080/nifi-registry-api/swagger/ui.html) |
| |
| 5) Accessing the application logs |
| |
| Logs will be available in `logs/nifi-registry-app.log` |
| |
| ### Database Testing |
| |
| In order to ensure that NiFi Registry works correctly against different relational databases, |
| the existing integration tests can be run against different databases by leveraging the [Testcontainers framework](https://www.testcontainers.org/). |
| |
| Spring profiles are used to control the DataSource factory that will be made available to the Spring application context. |
| DataSource factories are provided that use the Testcontainers framework to start a Docker container for a given database and create a corresponding DataSource. |
| If no profile is specified then an H2 DataSource will be used by default and no Docker containers are required. |
| |
| Assuming Docker is running on the system where the build is running, then the following commands can be run: |
| |
| | Target Database | Build Command | |
| |-----------------|--------------------------------------------------------------------| |
| | All supported | `mvn verify -Ptest-all-dbs` | |
| | H2 (default) | `mvn verify` | |
| | MariaDB 10.3 | `mvn verify -Pcontrib-check -Dspring.profiles.active=mariadb-10-3` | |
| | MySQL 8 | `mvn verify -Pcontrib-check -Dspring.profiles.active=mysql-8` | |
| | PostgreSQL 10 | `mvn verify -Dspring.profiles.active=postgres-10` | |
| |
| For a full list of the available DataSource factories, consult the `nifi-registry-test` module. |
| |
| ## Getting Help |
| If you have questions, you can reach out to our mailing list: dev@nifi.apache.org |
| ([archive](https://lists.apache.org/list.html?dev@nifi.apache.org)). For more interactive discussions, community members can often be found in the following locations: |
| |
| - Apache NiFi Slack Workspace: https://apachenifi.slack.com/ |
| |
| New users can join the workspace using the following [invite link](https://join.slack.com/t/apachenifi/shared_invite/zt-11njbtkdx-ZRU8FKYSWoEHRJetidy0zA). |
| |
| To submit a feature request or bug report, please file a Jira at [https://issues.apache.org/jira/projects/NIFI/issues](https://issues.apache.org/jira/projects/NIFI/issues). If this is a **security vulnerability report**, please email [security@nifi.apache.org](mailto:security@nifi.apache.org) directly and review the [Apache NiFi Security Vulnerability Disclosure](https://nifi.apache.org/security.html) and [Apache Software Foundation Security](https://www.apache.org/security/committers.html) processes first. |
| |
| ## Documentation |
| |
| See https://nifi.apache.org/ for the latest NiFi documentation. |
| |
| See https://nifi.apache.org/minifi and https://cwiki.apache.org/confluence/display/MINIFI for the latest MiNiFi-specific documentation. |
| |
| See https://nifi.apache.org/registry for the latest Registry-specific documentation. |
| |
| ## License |
| |
| Except as otherwise noted this software is licensed under the |
| [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| https://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| |
| ## Export Control |
| |
| This distribution includes cryptographic software. The country in which you |
| currently reside may have restrictions on the import, possession, use, and/or |
| re-export to another country, of encryption software. BEFORE using any |
| encryption software, please check your country's laws, regulations and |
| policies concerning the import, possession, or use, and re-export of encryption |
| software, to see if this is permitted. See <https://www.wassenaar.org/> for more |
| information. |
| |
| The U.S. Government Department of Commerce, Bureau of Industry and Security |
| (BIS), has classified this software as Export Commodity Control Number (ECCN) |
| 5D002.C.1, which includes information security software using or performing |
| cryptographic functions with asymmetric algorithms. The form and manner of this |
| Apache Software Foundation distribution makes it eligible for export under the |
| License Exception ENC Technology Software Unrestricted (TSU) exception (see the |
| BIS Export Administration Regulations, Section 740.13) for both object code and |
| source code. |
| |
| The following provides more details on the included cryptographic software: |
| |
| Apache NiFi uses BouncyCastle, JCraft Inc., and the built-in |
| Java cryptography libraries for SSL, SSH, and the protection |
| of sensitive configuration parameters. See |
| |
| - https://bouncycastle.org/about.html |
| - http://www.jcraft.com/c-info.html |
| - https://www.oracle.com/corporate/security-practices/corporate/governance/global-trade-compliance.html |
| |
| for more details on each of these libraries cryptography features. |
| |
| [nifi]: https://nifi.apache.org/ |
| [logo]: https://nifi.apache.org/assets/images/apache-nifi-logo.svg |
