| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| # Apache NiFi - MiNiFi - C++ C2 Readme. |
| |
| |
| This readme defines the Command and control configuration options that work with Apache NiFi. All |
| options defined are located in minifi.properties. |
| |
| ## Table of Contents |
| |
| - [Apache NiFi - MiNiFi - C++ C2 Readme.](#apache-nifi---minifi---c-c2-readme) |
| - [Table of Contents](#table-of-contents) |
| - [Description](#description) |
| - [Configuration](#configuration) |
| - [Base Options](#base-options) |
| - [Metrics](#metrics) |
| - [UpdatePolicies](#updatepolicies) |
| - [Triggers](#triggers) |
| - [C2 File triggers](#c2-file-triggers) |
| |
| ## Description |
| |
| Apache NiFi MiNiFi C++ can communicates with a C2 Server via HTTP REST protocol. The protocol |
| delivers a C2 response the server, expecting requests in a heartbeat response. The protocol transforms |
| the C2 messages into a protocol specific representation. The internal representation is an AST therefore |
| you must define the root classes, which configure the classes that branch from the root. You can define |
| arbitrary nodes and sub-trees, but this isn't necessary and only advantageous for custom C2 servers. That |
| will be explained in greater detail in the metrics section. |
| |
| ## Configuration |
| |
| ### Base Options |
| For more more insight into the API used within the C2 agent, please visit: |
| https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal |
| |
| Release 0.6.0: Please note that all c2 properties now exist as `nifi.c2.*`. If your configuration properties |
| files contain the former naming convention of `c2.*`, we will continue to support that as |
| an alternate key, but you are encouraged to switch your configuration options as soon as possible. |
| |
| Note: In release 0.8.0 there is a configuration option to minimize the heartbeat payload size by excluding agent manifest. |
| For that, add "nifi.c2.full.heartbeat"=false property. |
| With this change, heartbeat with agent manifest included is sent only for the first time then falls back to sending |
| light weight heartbeat. If for some reason the C2 server does not receive the first full heartbeat, the manifest can |
| be requested via C2 DESCRIBE manifest command. |
| |
| |
| #in minifi.properties |
| |
| # Disable/Enable C2 |
| nifi.c2.enable=true |
| |
| # specify which sections the heartbeat message should contain |
| # DeviceInfoNode: basic info about the system (OS, number of cores etc) |
| # AgentInformation: info about the MiNiFi agent, may include the manifest |
| # FlowInformation: information about the current flow, including queue sizes |
| # ConfigurationChecksums: hashes of the configuration files; can be used to detect unexpected modifications |
| # the default is |
| nifi.c2.root.classes=DeviceInfoNode,AgentInformation,FlowInformation |
| |
| # control c2 heartbeat interval |
| nifi.c2.agent.heartbeat.period=30 sec |
| |
| # enable reporter classes |
| nifi.c2.agent.heartbeat.reporter.classes=RESTReceiver |
| # If RESTReceiver is configured its listener port and optional SSL certificate can also be configured |
| nifi.c2.rest.listener.port=<port> |
| nifi.c2.rest.listener.cacert=<SSL Cert path> |
| |
| # specify the rest URIs if using RESTSender |
| nifi.c2.rest.url=http://<your-c2-server>/<c2-api-path>/c2-protocol/heartbeat |
| nifi.c2.rest.url.ack=http://<your-c2-server>/<c2-api-path>/c2-protocol/acknowledge |
| nifi.c2.flow.base.url=http://<your-c2-server>/<c2-api-path>/c2-protocol/ |
| |
| # c2 agent identifier -- must be defined to run agent |
| nifi.c2.agent.identifier=<your identifier> |
| |
| # c2 agent class -- must be defined to run agent |
| nifi.c2.agent.class=<your agent class> |
| |
| # configure SSL Context service for REST Protocol |
| #nifi.c2.rest.ssl.context.service |
| |
| # specify encoding strategy for c2 requests (gzip, none) |
| #nifi.c2.rest.request.encoding=none |
| |
| # minimize REST heartbeat updates |
| #nifi.c2.rest.heartbeat.minimize.updates=true |
| |
| #### Flow Id and URL |
| |
| Flow id and URL are usually retrieved from the C2 server. These identify the last updated flow version and where the flow was downloaded from. These properties are persisted in the minifi.properties file. |
| |
| # in minifi.properties |
| nifi.c2.flow.id=8da5de7f-dcdb-4f6b-aa2f-6f162a7f9dc4 |
| nifi.c2.flow.url=http://localhost:10090/c2-server/api/flows/8da5de7f-dcdb-4f6b-aa2f-6f162a7f9dc4/content?aid=efmtest |
| |
| #### Agent Identifier Fallback |
| |
| It is possible to set a persistent fallback agent id. This is needed so that the C2 server can identify the same agent after a restart, even if nifi.c2.agent.identifier is not specified. |
| |
| # in minifi.properties |
| nifi.c2.agent.identifier.fallback=my_fallback_id |
| |
| ### Metrics |
| |
| Command and Control metrics can be used to send metrics through the heartbeat or via the DESCRIBE |
| operation. Since responses are formed in an AST, metrics can be formed as a sub tree. Metrics classes |
| are defined apriori and may reference a metrics class specific to a processor. The following example describes |
| a configuration of an agent |
| |
| # in minifi.properties |
| |
| nifi.c2.root.class.definitions=metrics |
| nifi.c2.root.class.definitions.metrics.name=metrics |
| nifi.c2.root.class.definitions.metrics.metrics=runtimemetrics,loadmetrics,processorMetrics |
| nifi.c2.root.class.definitions.metrics.metrics.runtimemetrics.name=RuntimeMetrics |
| nifi.c2.root.class.definitions.metrics.metrics.runtimemetrics.classes=DeviceInfoNode,FlowInformation |
| nifi.c2.root.class.definitions.metrics.metrics.loadmetrics.name=LoadMetrics |
| nifi.c2.root.class.definitions.metrics.metrics.loadmetrics.classes=QueueMetrics,RepositoryMetrics |
| nifi.c2.root.class.definitions.metrics.metrics.processorMetrics.name=ProcessorMetric |
| nifi.c2.root.class.definitions.metrics.metrics.processorMetrics.classes=GetFileMetrics |
| |
| Processor metrics can also be configured using regular expressions with the `processorMetrics/` prefix, so the following definition is also valid: |
| |
| nifi.c2.root.class.definitions.metrics.metrics.processorMetrics.classes=processorMetrics/Get.*Metrics |
| |
| This example shows a metrics sub tree defined by the option 'nifi.c2.root.class.definitions'. |
| |
| This is a comma separated list of all sub trees. In the example, above, only one sub tree exists: metrics. |
| |
| The options below metrics define the sub-trees within metrics: typedmetrics and processorMetrics. Each of these has a name. |
| The classes sub option will define the metrics classes that are placed within this sub-tree. For the RESTProtocol, the above |
| configuration produces the following JSON: |
| |
| "metrics": { |
| "RuntimeMetrics": { |
| "deviceInfo": { |
| "systemInfo": { |
| "cpuLoadAverage": 2.59521484375, |
| "cpuUtilization": 0.1188118811881188, |
| "machineArch": "x86_64", |
| "memoryUsage": 13103550464, |
| "operatingSystem": "Linux", |
| "physicalMem": 67024097280, |
| "vCores": 12 |
| }, |
| "networkInfo": { |
| "hostname": "ggyimesi-5540-ubuntu", |
| "ipAddress": "192.168.50.181" |
| }, |
| "identifier": "13396751919892753964" |
| }, |
| "flowInfo": { |
| "versionedFlowSnapshotURI": { |
| "bucketId": "default", |
| "flowId": "8db40550-db5d-11ec-95d7-0433c2c9832b" |
| }, |
| "queues": { |
| "2438e3c8-015a-1000-79ca-83af40ec1997": { |
| "dataSize": 0, |
| "dataSizeMax": 1048576, |
| "name": "GetFile/success/LogAttribute", |
| "size": 0, |
| "sizeMax": 0, |
| "uuid": "2438e3c8-015a-1000-79ca-83af40ec1997" |
| } |
| }, |
| "components": { |
| "FlowController": { |
| "running": true, |
| "uuid": "2438e3c8-015a-1000-79ca-83af40ec1990" |
| }, |
| "GetFile": { |
| "running": false, |
| "uuid": "2438e3c8-015a-1000-79ca-83af40ec1991" |
| }, |
| "LogAttribute": { |
| "running": true, |
| "uuid": "2438e3c8-015a-1000-79ca-83af40ec1992" |
| } |
| }, |
| "flowId": "8db40550-db5d-11ec-95d7-0433c2c9832b" |
| } |
| }, |
| "LoadMetrics": { |
| "QueueMetrics": { |
| "GetFile/success/LogAttribute": { |
| "datasize": "0", |
| "datasizemax": "1048576", |
| "queued": "0", |
| "queuedmax": "0" |
| } |
| }, |
| "RepositoryMetrics": { |
| "ff": { |
| "full": false, |
| "running": false, |
| "size": "0" |
| }, |
| "repo_name": { |
| "full": false, |
| "running": true, |
| "size": "0" |
| } |
| } |
| }, |
| "ProcessorMetrics": { |
| "GetFileMetrics": { |
| "2438e3c8-015a-1000-79ca-83af40ec1991": { |
| "AcceptedFiles": 0, |
| "InputBytes": 0, |
| "OnTriggerInvocations": 0 |
| } |
| } |
| } |
| } |
| |
| ### UpdatePolicies |
| |
| Updates to MiNiFi C++ properties can be controlled through an UpdatePolicyControllerService named |
| C2UpdatePolicy. The service supports several configuration options. They are defined in the following example: |
| |
| Controller Services: |
| - id: 94491a38-015a-1000-0000-000000000001 |
| name: C2UpdatePolicy |
| class: UpdatePolicyControllerService |
| Properties: |
| # true enables all properties to be updated. |
| Allow All Properties: true |
| # allowed properties are those which can be updated |
| Allowed Properties: |
| - value: Property_1 |
| - value: Property_2 |
| Disallowed Properties: |
| - value: Property_3 |
| - value: Property_4 |
| |
| ### Triggers |
| |
| C2 Triggers can be activated to perform some C2 activity via a local event. Currently only FileUpdateTrigger exists, which monitors |
| for C2 File triggers to update the flow configuration. Classes can be defined as a comma separated list of classes to load via the option |
| nifi.c2.agent.trigger.classes |
| |
| |
| #### C2 File triggers |
| |
| C2 updates can be triggered with updates to a flow configuration file. It doesn't have to be the same base configuration file. It |
| will be copied into place. A new property, nifi.c2.file.watch, can be placed into minifi.properties to monitor. If the update time |
| changes while the agent is running, it will be copied into place of the file defined by nifi.flow.configuration.file. The agent |
| will then be restarted with the new flow configuration. If a failure occurs in reading that file or it is an invalid YAML file, the |
| update process will be halted. |
| |
| in minifi.properties to activate the file update trigger specify |
| |
| # specifying a trigger |
| nifi.c2.agent.trigger.classes=FileUpdateTrigger |
| nifi.c2.file.watch=<full path of file to monitor> |
